[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 506667025 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002009] Switched APIC routing to physical x2apic. [ 0.003011] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007008] pid_max: default: 32768 minimum: 301 [ 0.008131] LSM: Security Framework initializing [ 0.009035] Yama: becoming mindful. [ 0.010035] SELinux: Initializing. [ 0.011081] *** VALIDATE selinux *** [ 0.018676] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023247] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026106] *** VALIDATE tmpfs *** [ 0.028383] *** VALIDATE proc *** [ 0.029201] *** VALIDATE cgroup *** [ 0.030007] *** VALIDATE cgroup2 *** [ 0.031239] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034022] Spectre V2 : User space: Vulnerable [ 0.035007] Speculative Store Bypass: Vulnerable [ 0.038367] debug: unmapping init [mem 0xffffffff8ce59000-0xffffffff8ce60fff] [ 0.041109] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042658] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043018] ... version: 2 [ 0.044009] ... bit width: 48 [ 0.045008] ... generic registers: 4 [ 0.046010] ... value mask: 0000ffffffffffff [ 0.047009] ... max period: 00007fffffffffff [ 0.048009] ... fixed-purpose events: 3 [ 0.049008] ... event mask: 000000070000000f [ 0.051188] rcu: Hierarchical SRCU implementation. [ 0.053384] smp: Bringing up secondary CPUs ... [ 0.054581] x86: Booting SMP configuration: [ 0.055019] .... node #0, CPUs: #1 #2 #3 [ 0.061074] smp: Brought up 1 node, 4 CPUs [ 0.063011] smpboot: Max logical packages: 1 [ 0.064011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.102020] node 0 deferred pages initialised in 36ms [ 0.109443] devtmpfs: initialized [ 0.111337] x86/mm: Memory block size: 128MB [ 0.114234] gcov: version magic: 0x41383552 [ 0.118000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.121077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.125000] pinctrl core: initialized pinctrl subsystem [ 0.127255] [ 0.127939] ************************************************************* [ 0.131022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134013] ** ** [ 0.137016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.139012] ** ** [ 0.142017] ** This means that this kernel is built to expose internal ** [ 0.144014] ** IOMMU data structures, which may compromise security on ** [ 0.146012] ** your system. ** [ 0.148017] ** ** [ 0.150012] ** If you see this message and you are not debugging the ** [ 0.151010] ** kernel, report this immediately to your vendor! ** [ 0.153017] ** ** [ 0.156013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158011] ************************************************************* [ 0.163699] NET: Registered protocol family 16 [ 0.164463] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.167592] cpuidle: using governor menu [ 0.172007] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175363] PCI: Using configuration type 1 for base access [ 0.178125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189084] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.191013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195029] cryptd: max_cpu_qlen set to 1000 [ 0.198417] ACPI: Added _OSI(Module Device) [ 0.201011] ACPI: Added _OSI(Processor Device) [ 0.204013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213835] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220934] ACPI: Interpreter enabled [ 0.222057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223012] ACPI: Using IOAPIC for interrupt routing [ 0.225125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229382] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.242043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.245015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.249122] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254281] acpiphp: Slot [2] registered [ 0.256142] acpiphp: Slot [3] registered [ 0.257081] acpiphp: Slot [4] registered [ 0.259100] acpiphp: Slot [5] registered [ 0.262130] acpiphp: Slot [6] registered [ 0.263111] acpiphp: Slot [7] registered [ 0.265136] acpiphp: Slot [8] registered [ 0.267131] acpiphp: Slot [9] registered [ 0.268123] acpiphp: Slot [10] registered [ 0.270116] acpiphp: Slot [11] registered [ 0.271079] acpiphp: Slot [12] registered [ 0.273106] acpiphp: Slot [13] registered [ 0.274102] acpiphp: Slot [14] registered [ 0.276080] acpiphp: Slot [15] registered [ 0.277090] acpiphp: Slot [16] registered [ 0.279079] acpiphp: Slot [17] registered [ 0.280133] acpiphp: Slot [18] registered [ 0.282082] acpiphp: Slot [19] registered [ 0.283088] acpiphp: Slot [20] registered [ 0.285119] acpiphp: Slot [21] registered [ 0.286090] acpiphp: Slot [22] registered [ 0.288094] acpiphp: Slot [23] registered [ 0.290101] acpiphp: Slot [24] registered [ 0.291096] acpiphp: Slot [25] registered [ 0.293092] acpiphp: Slot [26] registered [ 0.294114] acpiphp: Slot [27] registered [ 0.296145] acpiphp: Slot [28] registered [ 0.298099] acpiphp: Slot [29] registered [ 0.299624] acpiphp: Slot [30] registered [ 0.301196] acpiphp: Slot [31] registered [ 0.303069] PCI host bridge to bus 0000:00 [ 0.304017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.307023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.312019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.315023] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.318028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.320163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.325843] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329547] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340853] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.345546] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355461] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.358844] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.361037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.364841] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.370016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.383014] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.390015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.393607] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.402024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.410029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.433020] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.444869] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.450022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482021] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.489000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.495021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.501023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.514020] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.524000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.528023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.531021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.544000] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.554760] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.571023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.586025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605025] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.620551] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.630020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.641026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.661026] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.672000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.677483] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.680479] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.682430] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.685258] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.686143] iommu: Default domain type: Passthrough [ 0.689317] SCSI subsystem initialized [ 0.691547] ACPI: bus type USB registered [ 0.696153] usbcore: registered new interface driver usbfs [ 0.697065] usbcore: registered new interface driver hub [ 0.698000] usbcore: registered new device driver usb [ 0.698000] pps_core: LinuxPPS API ver. 1 registered [ 0.700012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.702078] PTP clock support registered [ 0.704195] EDAC MC: Ver: 3.0.0 [ 0.706012] PCI: Using ACPI for IRQ routing [ 0.707000] NetLabel: Initializing [ 0.708013] NetLabel: domain hash size = 128 [ 0.710009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.712107] NetLabel: unlabeled traffic allowed by default [ 0.714343] vgaarb: loaded [ 0.716548] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.717009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.723170] clocksource: Switched to clocksource kvm-clock [ 0.847548] VFS: Disk quotas dquot_6.6.0 [ 0.848883] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.851264] *** VALIDATE ramfs *** [ 0.852460] *** VALIDATE hugetlbfs *** [ 0.854135] pnp: PnP ACPI init [ 0.856790] pnp: PnP ACPI: found 6 devices [ 0.874561] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.877763] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.879751] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.881514] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.884151] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.886422] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.890556] NET: Registered protocol family 2 [ 0.893321] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.901212] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.904816] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.910716] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.913873] TCP: Hash tables configured (established 65536 bind 65536) [ 0.917413] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.921513] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.924385] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.927146] NET: Registered protocol family 1 [ 0.931507] RPC: Registered named UNIX socket transport module. [ 0.934889] RPC: Registered udp transport module. [ 0.937678] RPC: Registered tcp transport module. [ 0.940097] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.942815] NET: Registered protocol family 44 [ 0.944969] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.948338] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.950263] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.952222] PCI: CLS 0 bytes, default 64 [ 0.953978] Unpacking initramfs... [ 2.642945] debug: unmapping init [mem 0xffff8edefcc54000-0xffff8edefffbffff] [ 2.649381] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.651232] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.654828] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.231328] Initialise system trusted keyrings [ 3.233029] Key type blacklist registered [ 3.236201] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.245589] zbud: loaded [ 3.249287] *** VALIDATE nfs *** [ 3.250438] *** VALIDATE nfs4 *** [ 3.252063] pstore: using deflate compression [ 3.255582] Platform Keyring initialized [ 3.513956] NET: Registered protocol family 38 [ 3.515789] Key type asymmetric registered [ 3.516953] Asymmetric key parser 'x509' registered [ 3.518538] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.528696] io scheduler mq-deadline registered [ 3.532469] io scheduler kyber registered [ 3.535583] io scheduler bfq registered [ 3.537456] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.541181] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.544291] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.547268] ACPI: Power Button [PWRF] [ 3.665153] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.773234] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.990235] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.083798] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.373982] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.410937] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.448780] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.453488] Non-volatile memory driver v1.3 [ 4.455220] Linux agpgart interface v0.103 [ 4.490525] virtio_blk virtio1: [vda] 133928 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.497988] vda: detected capacity change from 0 to 68571136 [ 4.518569] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.524517] vdb: detected capacity change from 0 to 1073741824 [ 4.541833] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.544639] vdc: detected capacity change from 0 to 2621440000 [ 4.562593] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.568775] vdd: detected capacity change from 0 to 2621440000 [ 4.585105] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.588069] vde: detected capacity change from 0 to 4294967296 [ 4.605956] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.609121] vdf: detected capacity change from 0 to 4294967296 [ 4.617117] libphy: Fixed MDIO Bus: probed [ 4.624880] usbcore: registered new interface driver usbserial_generic [ 4.628229] usbserial: USB Serial support registered for generic [ 4.633830] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.638451] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.639909] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.642737] mousedev: PS/2 mouse device common for all mice [ 4.649597] rtc_cmos 00:05: RTC can wake from S4 [ 4.652894] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.655830] rtc_cmos 00:05: registered as rtc0 [ 4.658732] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.665703] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.666917] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.673133] intel_pstate: CPU model not supported [ 4.675892] hid: raw HID events driver (C) Jiri Kosina [ 4.681909] usbcore: registered new interface driver usbhid [ 4.684089] usbhid: USB HID core driver [ 4.685382] drop_monitor: Initializing network drop monitor service [ 4.687477] Initializing XFRM netlink socket [ 4.689316] NET: Registered protocol family 10 [ 4.692493] Segment Routing with IPv6 [ 4.693703] NET: Registered protocol family 17 [ 4.696091] mpls_gso: MPLS GSO support [ 4.702475] RAS: Correctable Errors collector initialized. [ 4.704420] AVX version of gcm_enc/dec engaged. [ 4.705996] AES CTR mode by8 optimization enabled [ 4.855185] sched_clock: Marking stable (4855164582, 0)->(5885331055, -1030166473) [ 4.860808] registered taskstats version 1 [ 4.865424] Loading compiled-in X.509 certificates [ 4.867033] zswap: loaded using pool lzo/zbud [ 4.911246] Key type big_key registered [ 4.935409] Key type encrypted registered [ 4.936583] ima: No TPM chip found, activating TPM-bypass! [ 4.938571] ima: Allocated hash algorithm: sha1 [ 4.940455] ima: No architecture policies found [ 4.942314] evm: Initialising EVM extended attributes: [ 4.944363] evm: security.selinux [ 4.945663] evm: security.ima [ 4.946897] evm: security.capability [ 4.948348] evm: HMAC attrs: 0x1 [ 4.950862] rtc_cmos 00:05: setting system clock to 2025-12-03 06:49:50 UTC (1764744590) [ 4.958891] debug: unmapping init [mem 0xffffffff8de03000-0xffffffff8dffffff] [ 4.962520] debug: unmapping init [mem 0xffffffff8cb82000-0xffffffff8ce58fff] [ 4.971174] Write protecting the kernel read-only data: 28672k [ 4.976120] debug: unmapping init [mem 0xffffffff8b203000-0xffffffff8b3fffff] [ 4.978801] debug: unmapping init [mem 0xffffffff8bb14000-0xffffffff8bbfffff] [ 5.027416] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.038348] systemd[1]: Detected virtualization kvm. [ 5.040781] systemd[1]: Detected architecture x86-64. [ 5.042811] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.090525] systemd[1]: No hostname configured. [ 5.094881] systemd[1]: Set hostname to . [ 5.097105] random: systemd: uninitialized urandom read (16 bytes read) [ 5.099550] systemd[1]: Initializing machine ID from random generator. [ 5.204350] random: ln: uninitialized urandom read (6 bytes read) [ 5.441521] random: systemd: uninitialized urandom read (16 bytes read) [ 5.448375] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.454851] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.463755] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.065620] device-mapper: uevent: version 1.0.3 [ 7.067803] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook...[ 8.325137] random: fast init done [ 8.403554] virtio_net virtio0 ens2: renamed from eth0 [ 8.706584] scsi host0: ata_piix [ 8.917971] scsi host1: ata_piix [ 8.919025] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.920548] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.458664] random: crng init done [ 13.461935] random: 7 urandom warning(s) missed due to ratelimiting [ 14.063730] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.685921] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.701250] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.295268] SELinux: Disabled at runtime. [ 18.378186] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.392594] systemd[1]: Detected virtualization kvm. [ 18.394841] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.539607] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.545653] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.553465] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.562452] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.567851] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.585898] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.619777] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 20.088126] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 20.890326] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 21.857706] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.945137] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.459515] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.763050] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 26.692225] Key type dns_resolver registered [ 27.134209] NFS: Registering the id_resolver key type [ 27.136230] Key type id_resolver registered [ 27.139077] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg248-server login: [ 54.858749] libcfs: loading out-of-tree module taints kernel. [ 54.878929] Key type ._llcrypt registered [ 54.880472] Key type .llcrypt registered [ 54.928167] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_hostid [ 62.691586] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 63.272926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.280184] alg: No test for adler32 (adler32-zlib) [ 64.274704] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 64.581563] LNet: Added LNI 192.168.202.148@tcp [8/256/0/180] [ 66.192166] Key type lgssc registered [ 66.767725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.148242] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 74.536114] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 77.598620] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.403923] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.289285] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.590803] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing load_modules_local [ 94.612767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.638079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 94.647632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 95.752738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 95.774330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 95.825829] Lustre: lustre-MDT0000: new disk, initializing [ 95.870992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 95.881443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 97.407652] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.004680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.037419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 103.066353] Lustre: 6481:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 103.080966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 103.083292] Lustre: Skipped 1 previous similar message [ 103.127834] Lustre: lustre-MDT0001: new disk, initializing [ 103.153813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 103.165535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 103.170243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 104.571517] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.016445] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 110.697684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.731670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 110.848900] Lustre: lustre-OST0000: new disk, initializing [ 110.854990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 110.886865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 112.117754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 112.120666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 112.157515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 113.407157] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 119.198367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.224671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 119.264563] Lustre: lustre-OST0001: new disk, initializing [ 119.266786] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 119.287137] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 119.685234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 119.690714] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 119.705206] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 121.319694] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 127.566618] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 134.547927] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 141.943329] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing check_logdir /tmp/testlogs/ [ 143.810638] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing yml_node [ 145.969949] Lustre: DEBUG MARKER: Client: 2.17.0.RC1 [ 147.052778] Lustre: DEBUG MARKER: MDS: 2.17.0.RC1 [ 148.165899] Lustre: DEBUG MARKER: OSS: 2.17.0.RC1 [ 148.911253] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Wed Dec 3 01:52:13 EST 2025 [ 156.522901] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 157.294068] Lustre: DEBUG MARKER: === replay-single: start setup 01:52:21 (1764744741) === [ 158.795813] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing check_config_client /mnt/lustre [ 167.379898] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 168.854233] Lustre: 13153:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 170.262266] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.615041] Lustre: DEBUG MARKER: === replay-single: finish setup 01:52:36 (1764744756) === [ 173.404477] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 01:52:37 (1764744757) [ 176.890341] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 177.724782] Lustre: Failing over lustre-MDT0000 [ 177.798563] LustreError: 14024:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 177.828163] Lustre: server umount lustre-MDT0000 complete [ 180.192770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 180.198355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 181.218423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 181.225363] Lustre: Skipped 1 previous similar message [ 186.203570] LustreError: 14140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 186.210752] LustreError: 14140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 191.324083] LustreError: 6486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 191.335053] LustreError: 6486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 193.011174] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 193.013199] LDISKFS-fs (dm-0): recovery complete [ 193.025260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.040223] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 193.131740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.786470] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.444270] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 198.626154] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 198.639300] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 200.046677] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 200.797091] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 204.851556] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 01:53:09 (1764744789) [ 205.766647] Lustre: Failing over lustre-OST0000 [ 205.789073] LustreError: 15452:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 205.790946] LustreError: 15452:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 205.813856] Lustre: server umount lustre-OST0000 complete [ 206.693645] LustreError: 9455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 206.704190] LustreError: 9455:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 208.865589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 208.866091] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 208.871776] Lustre: Skipped 1 previous similar message [ 208.872780] LustreError: 8370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 208.883452] LustreError: 8370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 213.984779] LustreError: 8369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 213.991197] LustreError: 8369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 219.241283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 219.316665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.465384] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 220.790213] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 220.790286] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 220.799521] Lustre: Skipped 3 previous similar messages [ 221.722634] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.704129] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 226.505535] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 231.170829] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 01:53:35 (1764744815) [ 234.235518] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 235.258890] Lustre: Failing over lustre-MDT0000 [ 235.346690] LustreError: 17153:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 235.352440] LustreError: 17153:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 235.407091] Lustre: server umount lustre-MDT0000 complete [ 239.584714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 239.587269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 239.590650] Lustre: Skipped 1 previous similar message [ 239.593044] LustreError: 7431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 239.599511] LustreError: 7431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 251.356459] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 251.358240] LDISKFS-fs (dm-0): recovery complete [ 251.363339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 251.420096] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 251.515058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.059453] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.992427] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 253.996294] Lustre: lustre-MDT0000: Denying connection for new client 8ed89ac2-430c-4c44-8acb-7960471f2133 (at 192.168.202.48@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 256.995022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 256.998989] Lustre: Skipped 1 previous similar message [ 259.425219] Lustre: lustre-MDT0000: Denying connection for new client 8ed89ac2-430c-4c44-8acb-7960471f2133 (at 192.168.202.48@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 264.539787] Lustre: lustre-MDT0000: Denying connection for new client 8ed89ac2-430c-4c44-8acb-7960471f2133 (at 192.168.202.48@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 269.659829] Lustre: lustre-MDT0000: Denying connection for new client 8ed89ac2-430c-4c44-8acb-7960471f2133 (at 192.168.202.48@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 274.781693] Lustre: lustre-MDT0000: Denying connection for new client 8ed89ac2-430c-4c44-8acb-7960471f2133 (at 192.168.202.48@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 285.021079] Lustre: lustre-MDT0000: Denying connection for new client 8ed89ac2-430c-4c44-8acb-7960471f2133 (at 192.168.202.48@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 285.030135] Lustre: Skipped 1 previous similar message [ 305.498938] Lustre: lustre-MDT0000: Denying connection for new client 8ed89ac2-430c-4c44-8acb-7960471f2133 (at 192.168.202.48@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 305.506855] Lustre: Skipped 3 previous similar messages [ 323.500129] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 323.502180] Lustre: 17805:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 8c74b317-203f-4d91-823a-9e0d2160cbbd@ [ 323.506031] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 323.511079] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 323.511166] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 323.515765] Lustre: Skipped 2 previous similar messages [ 323.526937] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 323.526937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 329.845251] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 01:55:14 (1764744914) [ 332.797736] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 333.601263] Lustre: Failing over lustre-MDT0000 [ 333.665082] LustreError: 18862:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 333.667471] LustreError: 18862:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 333.692805] Lustre: server umount lustre-MDT0000 complete [ 333.793914] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.793961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 333.794164] LustreError: 10422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.794172] LustreError: 10422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 333.798747] Lustre: Skipped 4 previous similar messages [ 348.214618] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 348.216795] LDISKFS-fs (dm-0): recovery complete [ 348.222977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 348.270525] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 348.356383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.792555] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 350.578523] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 350.581292] Lustre: lustre-MDT0000: Denying connection for new client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 350.585825] Lustre: Skipped 3 previous similar messages [ 353.766162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 417.114859] Lustre: lustre-MDT0000: Denying connection for new client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 417.120205] Lustre: Skipped 12 previous similar messages [ 420.500127] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 420.502206] Lustre: 19515:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 8ed89ac2-430c-4c44-8acb-7960471f2133@ [ 420.505922] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 420.511337] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 420.511540] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 420.513888] Lustre: Skipped 2 previous similar messages [ 420.527976] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 420.527985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 425.977993] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 01:56:50 (1764745010) [ 428.455844] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 429.139958] Lustre: Failing over lustre-MDT0000 [ 429.200309] LustreError: 20572:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 429.202014] LustreError: 20572:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 429.226563] Lustre: server umount lustre-MDT0000 complete [ 430.560983] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.561249] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.564782] Lustre: Skipped 5 previous similar messages [ 430.569726] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 443.544508] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 443.546454] LDISKFS-fs (dm-0): recovery complete [ 443.551946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.590803] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.663769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.079138] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 447.835259] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 448.997284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 449.012296] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 449.034464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 449.034519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 450.395519] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 450.966291] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 454.433350] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 01:57:18 (1764745038) [ 456.973729] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 457.613961] Lustre: Failing over lustre-MDT0000 [ 457.802871] LustreError: 22392:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 457.804409] LustreError: 22392:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 457.832996] Lustre: server umount lustre-MDT0000 complete [ 459.232477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 459.232693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.239694] Lustre: Skipped 2 previous similar messages [ 472.072506] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 472.074312] LDISKFS-fs (dm-0): recovery complete [ 472.078091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.108426] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 473.434741] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 473.539079] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 477.697668] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 477.716202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 477.716226] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 478.930554] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 479.511816] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 483.048996] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 01:57:47 (1764745067) [ 485.588923] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 486.311826] Lustre: Failing over lustre-MDT0000 [ 486.437774] LustreError: 24213:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 486.439865] LustreError: 24213:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 486.466152] Lustre: server umount lustre-MDT0000 complete [ 487.904597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 498.144903] LustreError: 7431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.150692] LustreError: 7431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 500.730929] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 500.732432] LDISKFS-fs (dm-0): recovery complete [ 500.734902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.766980] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 502.160904] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 504.154781] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 506.338257] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 506.340337] Lustre: Skipped 7 previous similar messages [ 506.360421] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 506.376780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 506.376811] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 507.572957] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 508.099934] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 511.454481] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 01:58:15 (1764745095) [ 513.918859] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 514.589814] Lustre: Failing over lustre-MDT0000 [ 514.817643] Lustre: server umount lustre-MDT0000 complete [ 516.576834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.580124] Lustre: Skipped 8 previous similar messages [ 516.581269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 528.831391] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 528.832680] LDISKFS-fs (dm-0): recovery complete [ 528.835764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 528.870346] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.264597] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 534.019222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 534.019238] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 535.052494] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 535.558530] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 538.851502] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 01:58:43 (1764745123) [ 541.327880] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 541.945778] Lustre: Failing over lustre-MDT0000 [ 542.135533] LustreError: 27855:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 542.138048] LustreError: 27855:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 542.161385] Lustre: server umount lustre-MDT0000 complete [ 556.038696] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 556.040280] LDISKFS-fs (dm-0): recovery complete [ 556.043529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.354626] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 560.475049] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 560.478380] Lustre: Skipped 1 previous similar message [ 561.646495] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 561.650908] Lustre: Skipped 1 previous similar message [ 561.667343] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 561.667371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 562.783156] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 563.324586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 566.945403] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 01:59:11 (1764745151) [ 567.292729] Lustre: *** cfs_fail_loc=13b, val=315*** [ 567.294466] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 567.296371] LustreError: 6490:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf6d57b800 x1850468876800512/t38654705666(0) o35->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:613/0 lens 392/456 e 0 to 0 dl 1764745163 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 570.837808] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 571.495240] Lustre: Failing over lustre-MDT0000 [ 571.715758] Lustre: server umount lustre-MDT0000 complete [ 571.872994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 571.875483] LustreError: Skipped 1 previous similar message [ 585.831934] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 585.833581] LDISKFS-fs (dm-0): recovery complete [ 585.837595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.884954] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 585.889585] LustreError: Skipped 1 previous similar message [ 585.971550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.974347] Lustre: Skipped 4 previous similar messages [ 587.314301] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 591.330410] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 591.332633] Lustre: Skipped 11 previous similar messages [ 591.347707] Lustre: 6490:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf7625ea00 x1850468876800512/t38654705666(0) o35->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:637/0 lens 392/456 e 0 to 0 dl 1764745187 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 591.360673] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 591.360719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 592.532271] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 593.075016] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 596.452900] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 01:59:40 (1764745180) [ 598.941427] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 599.595136] Lustre: Failing over lustre-MDT0000 [ 599.694143] Lustre: server umount lustre-MDT0000 complete [ 601.568594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.572058] Lustre: Skipped 11 previous similar messages [ 613.995368] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 613.997236] LDISKFS-fs (dm-0): recovery complete [ 614.001273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.129109] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 615.511228] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 619.520896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 619.521105] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 620.592633] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 621.077639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 624.437983] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 02:00:08 (1764745208) [ 626.757702] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 627.057273] Lustre: *** cfs_fail_loc=114, val=0*** [ 627.906633] Lustre: Failing over lustre-MDT0000 [ 628.018849] LustreError: 33462:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 628.021301] LustreError: 33462:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 628.043942] Lustre: server umount lustre-MDT0000 complete [ 629.730207] LustreError: 14140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.734784] LustreError: 14140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 62 previous similar messages [ 641.900637] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 641.902009] LDISKFS-fs (dm-0): recovery complete [ 641.905241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.006833] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 642.394607] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 642.398378] Lustre: Skipped 2 previous similar messages [ 643.269580] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 647.149668] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 647.153966] Lustre: Skipped 2 previous similar messages [ 647.172369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 647.172384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 648.337093] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 648.879424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 652.170920] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 02:00:36 (1764745236) [ 654.775907] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 655.097095] Lustre: *** cfs_fail_loc=128, val=0*** [ 656.056733] Lustre: Failing over lustre-MDT0000 [ 656.246094] Lustre: server umount lustre-MDT0000 complete [ 657.376859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 657.378736] LustreError: Skipped 2 previous similar messages [ 670.291721] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 670.293104] LDISKFS-fs (dm-0): recovery complete [ 670.295889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.323802] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.328132] LustreError: Skipped 2 previous similar messages [ 670.401778] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 671.644482] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 675.839918] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 675.839962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 677.029188] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 677.586442] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 680.934949] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 02:01:05 (1764745265) [ 683.329492] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 683.978751] Lustre: Failing over lustre-MDT0000 [ 684.116517] Lustre: server umount lustre-MDT0000 complete [ 698.219567] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 698.220798] LDISKFS-fs (dm-0): recovery complete [ 698.223994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.335352] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 699.603140] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 703.516913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 703.516925] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 704.630991] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 705.152714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 708.608922] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 02:01:33 (1764745293) [ 711.276569] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 711.956163] Lustre: Failing over lustre-MDT0000 [ 712.101142] Lustre: server umount lustre-MDT0000 complete [ 725.984611] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 725.985667] LDISKFS-fs (dm-0): recovery complete [ 725.988059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.094178] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 727.346920] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 731.112167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 731.113988] Lustre: Skipped 19 previous similar messages [ 731.155083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 731.155126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 732.442872] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 732.975210] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 736.297516] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 02:02:00 (1764745320) [ 738.818983] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 740.797219] Lustre: Failing over lustre-MDT0000 [ 740.897757] Lustre: server umount lustre-MDT0000 complete [ 741.344480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.349030] Lustre: Skipped 19 previous similar messages [ 754.939831] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 754.941703] LDISKFS-fs (dm-0): recovery complete [ 754.944567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 755.051498] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 756.259049] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 760.818843] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 760.818894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 761.932166] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 762.504079] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 771.671619] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 02:02:36 (1764745356) [ 774.163758] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 774.797349] Lustre: Failing over lustre-MDT0000 [ 774.986546] LustreError: 42699:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 774.988719] LustreError: 42699:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 775.010865] Lustre: server umount lustre-MDT0000 complete [ 788.825468] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 788.826768] LDISKFS-fs (dm-0): recovery complete [ 788.829304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 788.929275] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 790.124911] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 791.898813] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 791.902238] Lustre: Skipped 4 previous similar messages [ 794.095799] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 794.099313] Lustre: Skipped 4 previous similar messages [ 794.113084] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 794.113123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 795.205282] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 795.732300] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 801.144623] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 02:03:05 (1764745385) [ 803.766215] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 804.426048] Lustre: Failing over lustre-MDT0000 [ 804.655720] Lustre: server umount lustre-MDT0000 complete [ 809.440515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 809.443937] LustreError: Skipped 3 previous similar messages [ 818.814267] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 818.815427] LDISKFS-fs (dm-0): recovery complete [ 818.817723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.848772] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 818.851770] LustreError: Skipped 4 previous similar messages [ 820.269587] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 824.321789] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 824.321828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 825.619105] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 826.202664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 829.644333] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 02:03:34 (1764745414) [ 832.222458] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 832.867957] Lustre: Failing over lustre-MDT0000 [ 833.133854] Lustre: server umount lustre-MDT0000 complete [ 847.471276] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 847.473167] LDISKFS-fs (dm-0): recovery complete [ 847.476718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.582699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 847.584972] Lustre: Skipped 8 previous similar messages [ 847.595306] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 847.597558] Lustre: Skipped 1 previous similar message [ 848.964940] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 852.991304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 852.991315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 854.197992] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 854.751710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 858.229909] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 02:04:02 (1764745442) [ 860.774802] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 861.487104] Lustre: Failing over lustre-MDT0000 [ 861.745884] Lustre: server umount lustre-MDT0000 complete [ 876.030264] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 876.032172] LDISKFS-fs (dm-0): recovery complete [ 876.034966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.551642] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 881.673520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 881.673537] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 882.855224] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 883.432147] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 886.928259] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 02:04:31 (1764745471) [ 889.524513] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 890.210926] Lustre: Failing over lustre-MDT0000 [ 890.394957] Lustre: server umount lustre-MDT0000 complete [ 891.873383] LustreError: 14140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 891.878202] LustreError: 14140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 126 previous similar messages [ 904.612371] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 904.613544] LDISKFS-fs (dm-0): recovery complete [ 904.616255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.156956] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 909.824466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 909.824470] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 911.083857] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 911.684057] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 915.229227] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 02:04:59 (1764745499) [ 917.681917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 918.299902] Lustre: Failing over lustre-MDT0000 [ 918.488597] Lustre: server umount lustre-MDT0000 complete [ 932.534506] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 932.535726] LDISKFS-fs (dm-0): recovery complete [ 932.540465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.655891] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 932.657617] Lustre: Skipped 2 previous similar messages [ 934.034130] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 937.986309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 937.986317] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 939.206393] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 939.871263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 943.442548] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 02:05:27 (1764745527) [ 946.126610] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 946.808214] Lustre: Failing over lustre-MDT0000 [ 947.069041] Lustre: server umount lustre-MDT0000 complete [ 961.334883] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 961.336910] LDISKFS-fs (dm-0): recovery complete [ 961.340756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.829770] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 966.663348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 966.663359] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 967.800970] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 968.306025] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 971.608749] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 02:05:56 (1764745556) [ 974.047698] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 974.674489] Lustre: Failing over lustre-MDT0000 [ 974.886695] Lustre: server umount lustre-MDT0000 complete [ 989.011324] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 989.012919] LDISKFS-fs (dm-0): recovery complete [ 989.016043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.518732] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 994.275114] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 994.278072] Lustre: Skipped 35 previous similar messages [ 994.300035] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 994.300051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 995.517972] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 996.088884] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 999.568050] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 02:06:24 (1764745584) [ 1002.170331] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1002.803153] Lustre: Failing over lustre-MDT0000 [ 1003.068069] Lustre: server umount lustre-MDT0000 complete [ 1004.512928] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.518306] Lustre: Skipped 35 previous similar messages [ 1017.307370] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1017.308604] LDISKFS-fs (dm-0): recovery complete [ 1017.312416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.820739] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1022.456581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1022.456583] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1023.627567] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1024.149824] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1027.754346] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 02:06:52 (1764745612) [ 1030.430939] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1031.123460] Lustre: Failing over lustre-MDT0000 [ 1031.242663] LustreError: 59092:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1031.244722] LustreError: 59092:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 1031.268101] Lustre: server umount lustre-MDT0000 complete [ 1045.457049] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1045.458747] LDISKFS-fs (dm-0): recovery complete [ 1045.461415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.072282] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1047.899954] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1047.903360] Lustre: Skipped 8 previous similar messages [ 1050.604819] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1050.607976] Lustre: Skipped 8 previous similar messages [ 1050.625995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 1050.626026] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:897) [ 1051.879376] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1052.476194] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1055.920229] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 02:07:20 (1764745640) [ 1058.477472] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1059.162219] Lustre: Failing over lustre-MDT0000 [ 1059.357887] Lustre: server umount lustre-MDT0000 complete [ 1073.624679] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1073.625976] LDISKFS-fs (dm-0): recovery complete [ 1073.629300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.759929] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1073.761551] Lustre: Skipped 4 previous similar messages [ 1075.155394] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1078.783632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1078.783694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1079.936835] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1080.510316] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1084.072104] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 02:07:48 (1764745668) [ 1086.619362] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1087.289063] Lustre: Failing over lustre-MDT0000 [ 1087.544229] Lustre: server umount lustre-MDT0000 complete [ 1088.993090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1088.995527] LustreError: Skipped 8 previous similar messages [ 1101.784836] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1101.785989] LDISKFS-fs (dm-0): recovery complete [ 1101.788749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1101.824664] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1101.827384] LustreError: Skipped 9 previous similar messages [ 1103.256345] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1106.941088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 1106.941114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 1108.123614] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1108.752487] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1112.298385] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 02:08:16 (1764745696) [ 1114.858530] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1115.501974] Lustre: Failing over lustre-MDT0000 [ 1115.723801] Lustre: server umount lustre-MDT0000 complete [ 1129.867779] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1129.869090] LDISKFS-fs (dm-0): recovery complete [ 1129.872793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.282869] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1135.098749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 1135.098776] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:993) [ 1136.312404] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1136.851761] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1140.274648] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 02:08:44 (1764745724) [ 1142.873056] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1143.563727] Lustre: Failing over lustre-MDT0000 [ 1143.801849] Lustre: server umount lustre-MDT0000 complete [ 1158.158697] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1158.159970] LDISKFS-fs (dm-0): recovery complete [ 1158.163415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1159.733585] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1163.780746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1163.780756] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1164.967259] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1165.549280] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1169.074445] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 02:09:13 (1764745753) [ 1171.586425] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1172.301628] Lustre: Failing over lustre-MDT0000 [ 1172.566577] Lustre: server umount lustre-MDT0000 complete [ 1187.051906] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1187.053501] LDISKFS-fs (dm-0): recovery complete [ 1187.057091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.570435] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1192.449749] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1192.449749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1193.733851] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1194.296333] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1197.848459] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 02:09:42 (1764745782) [ 1200.540337] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1201.254433] Lustre: Failing over lustre-MDT0000 [ 1201.453437] Lustre: server umount lustre-MDT0000 complete [ 1215.821108] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1215.822509] LDISKFS-fs (dm-0): recovery complete [ 1215.825970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.350306] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1221.116540] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 1221.116602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 1222.445647] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1223.029436] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1226.579759] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 02:10:10 (1764745810) [ 1227.918042] Lustre: 71490:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b9ec3cac-96a0-43fb-a74c-124f16871d0e at adminstrative request [ 1231.547986] Lustre: Failing over lustre-MDT0000 [ 1231.702466] Lustre: server umount lustre-MDT0000 complete [ 1244.660503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1246.138446] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1250.295327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1250.295354] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 1251.439136] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1251.959548] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1254.251658] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1256.879262] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1258.990735] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 02:10:43 (1764745843) [ 1259.333898] Lustre: 73332:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b9ec3cac-96a0-43fb-a74c-124f16871d0e at adminstrative request [ 1264.218730] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 02:10:48 (1764745848) [ 1266.875499] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1267.618981] Lustre: Failing over lustre-MDT0000 [ 1267.823188] Lustre: server umount lustre-MDT0000 complete [ 1281.967353] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1281.969328] LDISKFS-fs (dm-0): recovery complete [ 1281.971832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1283.379884] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1287.188632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 1287.188632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 1288.360747] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1288.927805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1292.426219] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 02:11:16 (1764745876) [ 1295.009942] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1295.693166] Lustre: Failing over lustre-MDT0000 [ 1295.927281] Lustre: server umount lustre-MDT0000 complete [ 1312.353297] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1312.355278] LDISKFS-fs (dm-0): recovery complete [ 1312.362875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.762799] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1317.924314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1317.924457] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1319.746951] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1320.465667] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1323.724849] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 02:11:48 (1764745908) [ 1326.142478] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1326.782110] Lustre: Failing over lustre-MDT0000 [ 1326.925911] Lustre: server umount lustre-MDT0000 complete [ 1340.729362] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1340.730262] LDISKFS-fs (dm-0): recovery complete [ 1340.732511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1340.827648] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1340.829218] Lustre: Skipped 8 previous similar messages [ 1342.118492] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1346.047565] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1346.050191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1347.188901] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1347.714183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1351.108644] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 02:12:15 (1764745935) [ 1353.728384] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1354.367924] Lustre: Failing over lustre-MDT0000 [ 1354.605912] Lustre: server umount lustre-MDT0000 complete [ 1368.682380] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1368.683555] LDISKFS-fs (dm-0): recovery complete [ 1368.686469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.781476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.783347] Lustre: Skipped 17 previous similar messages [ 1370.026717] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1374.197974] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1374.197978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1375.302712] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1375.818059] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1379.128349] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 02:12:43 (1764745963) [ 1381.543573] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1382.186270] Lustre: Failing over lustre-MDT0000 [ 1382.408592] Lustre: server umount lustre-MDT0000 complete [ 1396.706090] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1396.707700] LDISKFS-fs (dm-0): recovery complete [ 1396.711030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1398.198613] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1401.847145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 1401.847155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 1403.140256] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1403.677272] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1407.036678] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 02:13:11 (1764745991) [ 1409.461164] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1410.139604] Lustre: Failing over lustre-MDT0000 [ 1410.337381] Lustre: server umount lustre-MDT0000 complete [ 1411.419797] LustreError: 8071:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1411.427982] LustreError: 8071:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 262 previous similar messages [ 1424.368376] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1424.369863] LDISKFS-fs (dm-0): recovery complete [ 1424.372813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.771533] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1429.507756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1429.507772] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1430.711366] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1431.325928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1434.736887] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 02:13:39 (1764746019) [ 1437.209637] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1437.883019] Lustre: Failing over lustre-MDT0000 [ 1438.032411] Lustre: server umount lustre-MDT0000 complete [ 1451.952502] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1451.954621] LDISKFS-fs (dm-0): recovery complete [ 1451.957213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.269285] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1457.155525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1457.155532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1458.283639] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1458.800286] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1462.181075] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 02:14:06 (1764746046) [ 1464.770199] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1465.411425] Lustre: Failing over lustre-MDT0000 [ 1465.551302] Lustre: server umount lustre-MDT0000 complete [ 1479.934645] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1479.936972] LDISKFS-fs (dm-0): recovery complete [ 1479.940837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.507776] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1485.307781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1485.307781] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1486.496614] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1487.042615] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1490.454477] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 02:14:34 (1764746074) [ 1493.125030] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1493.811186] Lustre: Failing over lustre-MDT0000 [ 1493.991197] Lustre: server umount lustre-MDT0000 complete [ 1508.464174] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1508.465823] LDISKFS-fs (dm-0): recovery complete [ 1508.468684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.944826] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1513.958093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1513.961541] Lustre: Skipped 71 previous similar messages [ 1513.983875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1513.983881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1515.215337] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1515.802441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1519.281558] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 02:15:03 (1764746103) [ 1522.024085] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1522.688965] Lustre: Failing over lustre-MDT0000 [ 1522.844813] Lustre: server umount lustre-MDT0000 complete [ 1524.193045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.199261] Lustre: Skipped 71 previous similar messages [ 1536.943980] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1536.945557] LDISKFS-fs (dm-0): recovery complete [ 1536.947897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.314890] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1542.133682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1542.133682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1543.253350] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1543.799252] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1547.189434] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 02:15:31 (1764746131) [ 1549.625546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1550.226949] Lustre: Failing over lustre-MDT0000 [ 1550.426274] LustreError: 92228:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1550.428787] LustreError: 92228:0:(obd_class.h:479:obd_check_dev()) Skipped 143 previous similar messages [ 1550.452764] Lustre: server umount lustre-MDT0000 complete [ 1564.568319] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1564.570145] LDISKFS-fs (dm-0): recovery complete [ 1564.574894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.019777] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1565.022800] Lustre: Skipped 17 previous similar messages [ 1565.908715] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1569.770718] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1569.772895] Lustre: Skipped 17 previous similar messages [ 1569.787427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 1569.787444] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 1570.871974] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1571.382179] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1574.607346] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 02:15:59 (1764746159) [ 1574.938199] Lustre: 93605:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b9ec3cac-96a0-43fb-a74c-124f16871d0e at adminstrative request [ 1578.244299] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 02:16:02 (1764746162) [ 1580.462151] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1581.113497] Lustre: Failing over lustre-MDT0000 [ 1581.299423] Lustre: server umount lustre-MDT0000 complete [ 1585.399211] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1585.400165] LDISKFS-fs (dm-0): recovery complete [ 1585.402495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.513455] Lustre: lustre-MDT0000: Aborting client recovery [ 1585.515669] LustreError: 94884:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1585.519054] Lustre: 94917:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1585.522969] Lustre: 94917:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client b9ec3cac-96a0-43fb-a74c-124f16871d0e@ [ 1585.527598] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1585.531098] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1585.534516] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1585.553980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 1585.553980] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 1586.845365] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1590.756043] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1598.334510] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 02:16:22 (1764746182) [ 1600.777383] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1601.373341] Lustre: Failing over lustre-MDT0000 [ 1601.524904] Lustre: server umount lustre-MDT0000 complete [ 1605.667237] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1605.669021] LDISKFS-fs (dm-0): recovery complete [ 1605.672640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.790484] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1605.823970] Lustre: lustre-MDT0000: Aborting client recovery [ 1605.825871] LustreError: 96729:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1605.829270] Lustre: 96762:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1605.832776] Lustre: 96762:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1605.835860] Lustre: 96762:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1605.838416] Lustre: 96762:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1605.841195] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1605.844021] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 1605.850808] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 1605.872201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1537) [ 1605.872201] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1537) [ 1607.207380] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1611.236472] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1616.476703] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1616.478961] Lustre: Skipped 1 previous similar message [ 1618.497707] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 02:16:42 (1764746202) [ 1620.995890] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1621.642239] Lustre: Failing over lustre-MDT0000 [ 1621.724466] Lustre: server umount lustre-MDT0000 complete [ 1625.833202] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1625.834283] LDISKFS-fs (dm-0): recovery complete [ 1625.836816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.870964] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1625.874517] LustreError: Skipped 18 previous similar messages [ 1626.001738] Lustre: lustre-MDT0000: Aborting client recovery [ 1626.003228] LustreError: 98577:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1626.005327] Lustre: 98610:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1626.007563] Lustre: 98610:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1626.009514] Lustre: 98610:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1626.012336] Lustre: 98610:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1626.015979] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1626.018811] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 1626.024647] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 1626.045059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 1626.046321] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 1627.265980] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1631.203572] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1638.667942] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 02:17:03 (1764746223) [ 1639.015855] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1639.017394] LustreError: 17789:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf6cf3ed80 x1850468877726208/t201863462916(0) o36->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:175/0 lens 512/456 e 0 to 0 dl 1764746235 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1641.626124] Lustre: Failing over lustre-MDT0000 [ 1641.715831] Lustre: server umount lustre-MDT0000 complete [ 1644.855798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1645.004313] Lustre: lustre-MDT0000: Aborting client recovery [ 1645.005578] LustreError: 99884:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1645.007591] Lustre: 99917:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1645.010382] Lustre: 99917:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1645.014410] Lustre: 99917:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1645.018051] Lustre: 99917:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1645.019925] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1645.022977] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 1645.029853] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 1645.050782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 1645.050837] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1601) [ 1646.522225] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1650.145879] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1658.189684] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1658.740976] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 02:17:23 (1764746243) [ 1661.388359] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1662.368306] Lustre: Failing over lustre-MDT0000 [ 1662.503896] Lustre: server umount lustre-MDT0000 complete [ 1665.505345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1665.508447] LustreError: Skipped 17 previous similar messages [ 1667.003550] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1667.004797] LDISKFS-fs (dm-0): recovery complete [ 1667.007297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.146493] Lustre: lustre-MDT0000: Aborting client recovery [ 1667.148010] LustreError: 101827:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1667.151193] Lustre: 101860:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1667.154082] Lustre: 101860:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1667.155939] Lustre: 101860:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1667.158641] Lustre: 101860:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1667.161955] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1667.165075] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 1667.171118] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 1667.195705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 1667.195841] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1633) [ 1668.590316] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1672.161946] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1680.424351] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 02:17:44 (1764746264) [ 1688.924500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1689.581031] Lustre: Failing over lustre-MDT0000 [ 1689.589251] LustreError: 3626:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8edf58fcc000 x1850468887430400/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 1689.860331] Lustre: server umount lustre-MDT0000 complete [ 1704.197794] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1704.199134] LDISKFS-fs (dm-0): recovery complete [ 1704.201766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.733728] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1709.578446] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 1709.578446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 1710.841681] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1711.434103] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1719.645654] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 02:18:24 (1764746304) [ 1726.210761] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1729.014930] Lustre: Failing over lustre-MDT0000 [ 1729.220209] Lustre: server umount lustre-MDT0000 complete [ 1743.500027] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1743.501208] LDISKFS-fs (dm-0): recovery complete [ 1743.503699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1745.013842] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1749.577236] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 1749.577237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 1750.764455] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1751.308511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1759.368375] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 02:19:03 (1764746343) [ 1760.108438] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1760.439493] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1760.443496] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1762.595547] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 02:19:06 (1764746346) [ 1768.921332] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1772.040864] Lustre: Failing over lustre-OST0000 [ 1772.100780] Lustre: server umount lustre-OST0000 complete [ 1786.499224] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 1786.500852] LDISKFS-fs (dm-2): recovery complete [ 1786.503767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1788.497227] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1833.823674] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 02:20:18 (1764746418) [ 1836.578481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1837.542344] Lustre: Failing over lustre-MDT0000 [ 1837.699971] Lustre: server umount lustre-MDT0000 complete [ 1851.944080] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1851.945729] LDISKFS-fs (dm-0): recovery complete [ 1851.948508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.328684] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1857.529864] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 1857.529905] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 1858.670647] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1859.221613] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1872.785715] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 02:20:57 (1764746457) [ 1873.889115] Lustre: 110919:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764746443/real 1764746443] req@ffff8ede490a0700 x1850468887880320/t0(0) o5->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1764746459 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-0-0.0' uid:0 gid:0 projid:4294967295 [ 1873.897869] LustreError: 110919:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 1873.899275] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1874.458340] LustreError: 6488:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1874.912533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 1879.520106] LustreError: 6488:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1879.522091] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnecting [ 1880.132024] LustreError: 6488:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1885.152172] LustreError: 6488:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1885.155526] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnecting [ 1885.796107] LustreError: 6488:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1891.296147] LustreError: 6488:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1891.299557] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnecting [ 1891.954071] LustreError: 6488:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1897.440134] LustreError: 6488:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1898.059321] LustreError: 6488:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1903.072115] LustreError: 6488:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1903.076408] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnecting [ 1903.080718] Lustre: Skipped 1 previous similar message [ 1909.810850] LustreError: 17789:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1909.813094] LustreError: 17789:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 1914.848091] LustreError: 17789:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1914.853317] LustreError: 17789:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 1920.992157] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnecting [ 1920.994770] Lustre: Skipped 2 previous similar messages [ 1927.769498] LustreError: 17789:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1927.771486] LustreError: 17789:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 1933.280149] LustreError: 17789:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1933.284600] LustreError: 17789:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 1936.163833] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 02:22:00 (1764746520) [ 1936.826326] LustreError: 17789:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1946.971309] Lustre: lustre-MDT0000: Export ffff8edf7680d000 already connecting from 192.168.202.48@tcp [ 1952.090883] Lustre: lustre-MDT0000: Export ffff8edf7680d000 already connecting from 192.168.202.48@tcp [ 1957.210996] Lustre: lustre-MDT0000: Export ffff8edf7680d000 already connecting from 192.168.202.48@tcp [ 1962.331083] Lustre: lustre-MDT0000: Export ffff8edf7680d000 already connecting from 192.168.202.48@tcp [ 1962.332899] Lustre: Skipped 1 previous similar message [ 1967.457254] Lustre: lustre-MDT0000: Export ffff8edf7680d000 already connecting from 192.168.202.48@tcp [ 1976.880135] LustreError: 17789:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 1976.885786] Lustre: 17789:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8ede4900fb80 x1850468880356736/t0(0) o38->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:0/0 lens 520/416 e 0 to 0 dl 1764746542 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 1977.696624] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnecting [ 1977.703671] Lustre: Skipped 3 previous similar messages [ 1977.706547] LustreError: 8071:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2002.275781] Lustre: lustre-MDT0000: Export ffff8edf7680d000 already connecting from 192.168.202.48@tcp [ 2002.284098] Lustre: Skipped 1 previous similar message [ 2017.720183] LustreError: 8071:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2017.725555] Lustre: 8071:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8edf7bf2df80 x1850468880361984/t0(0) o38->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:0/0 lens 520/416 e 0 to 0 dl 1764746583 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2022.415168] LustreError: 8071:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2047.323267] Lustre: lustre-MDT0000: Export ffff8edf7680d000 already connecting from 192.168.202.48@tcp [ 2047.326401] Lustre: Skipped 3 previous similar messages [ 2062.464803] LustreError: 8071:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2062.474833] Lustre: 8071:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff8ede46a9dc00 x1850468880365568/t0(0) o38->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:0/0 lens 520/416 e 0 to 0 dl 1764746627 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2062.683253] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnecting [ 2062.687368] Lustre: Skipped 1 previous similar message [ 2062.689148] LustreError: 14140:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2087.261231] Lustre: lustre-MDT0000: Export ffff8edf7680d000 already connecting from 192.168.202.48@tcp [ 2087.265101] Lustre: Skipped 2 previous similar messages [ 2102.752241] LustreError: 14140:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2102.755689] Lustre: 14140:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8edf6c82ce00 x1850468880368512/t0(0) o38->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:0/0 lens 520/416 e 0 to 0 dl 1764746668 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2107.740072] LustreError: 6487:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2147.792131] LustreError: 6487:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2147.795558] Lustre: 6487:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8edf78bb2680 x1850468880372224/t0(0) o38->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:0/0 lens 520/416 e 0 to 0 dl 1764746713 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2152.046800] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 02:25:36 (1764746736) [ 2155.142866] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2156.477520] Lustre: Failing over lustre-MDT0000 [ 2156.549599] LustreError: 115008:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2156.552821] LustreError: 115008:0:(obd_class.h:479:obd_check_dev()) Skipped 75 previous similar messages [ 2156.592917] Lustre: server umount lustre-MDT0000 complete [ 2157.916088] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.921758] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 150 previous similar messages [ 2159.584649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2159.589596] Lustre: Skipped 40 previous similar messages [ 2160.956516] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2160.957884] LDISKFS-fs (dm-0): recovery complete [ 2160.961139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.086991] Lustre: *** cfs_fail_loc=712, val=0*** [ 2161.089239] LustreError: 42256:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8ede44731f80 x1850468888028032/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2161.105980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2161.108864] Lustre: Skipped 16 previous similar messages [ 2161.155916] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2161.156070] Lustre: lustre-MDT0000: Aborting client recovery [ 2161.157880] Lustre: Skipped 27 previous similar messages [ 2161.159186] LustreError: 115652:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2161.164123] Lustre: 115685:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2161.168017] Lustre: 115685:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2161.170443] Lustre: 115685:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2161.174639] Lustre: 115685:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2161.177470] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2161.180369] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2161.186686] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2161.208209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2161.208214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2162.777733] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2166.242816] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2166.244877] Lustre: Skipped 47 previous similar messages [ 2166.246959] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2173.366403] Lustre: Failing over lustre-MDT0000 [ 2173.560734] Lustre: server umount lustre-MDT0000 complete [ 2187.607345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2188.636071] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2188.641142] Lustre: Skipped 4 previous similar messages [ 2189.584292] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2192.874377] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2192.877857] Lustre: Skipped 4 previous similar messages [ 2192.904723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2192.904723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2194.423654] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2195.239069] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2199.181714] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 02:26:23 (1764746783) [ 2199.226448] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnecting [ 2199.229637] Lustre: Skipped 2 previous similar messages [ 2202.635579] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 02:26:26 (1764746786) [ 2203.042294] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2203.044252] LustreError: 6495:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf6d57b100 x1850468880428032/t0(0) o700->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:739/0 lens 264/248 e 0 to 0 dl 1764746799 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2219.557577] Lustre: Failing over lustre-MDT0000 [ 2219.712314] Lustre: server umount lustre-MDT0000 complete [ 2233.298920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.381452] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.384333] LustreError: Skipped 7 previous similar messages [ 2235.066365] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2238.973365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2238.973522] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2240.432959] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2241.150798] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2245.661443] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 02:27:10 (1764746830) [ 2246.645165] Lustre: Failing over lustre-OST0000 [ 2246.707394] Lustre: server umount lustre-OST0000 complete [ 2260.457729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2263.108962] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2266.680230] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2267.371143] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2333.552224] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 02:28:37 (1764746917) [ 2336.713080] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2337.665844] Lustre: Failing over lustre-MDT0000 [ 2337.763221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.771029] Lustre: Skipped 3 previous similar messages [ 2337.881726] Lustre: server umount lustre-MDT0000 complete [ 2352.912803] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2352.913854] LDISKFS-fs (dm-0): recovery complete [ 2352.918578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2354.640627] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2358.468063] Lustre: *** cfs_fail_loc=216, val=0*** [ 2358.469750] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2358.471099] LustreError: 121458:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2359.521635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2421.960703] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 02:30:06 (1764747006) [ 2430.602882] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 02:30:14 (1764747014) [ 2431.873278] Lustre: Failing over lustre-MDT0000 [ 2432.045574] Lustre: server umount lustre-MDT0000 complete [ 2445.620501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.338587] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2450.918831] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2450.921237] LustreError: 123027:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf6cf8ce00 x1850468880547712/t0(0) o101->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:276/0 lens 328/344 e 0 to 0 dl 1764747091 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2510.700787] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnected, waiting for 2 clients in recovery for 1:04 [ 2510.714587] Lustre: 123027:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2510.749844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 2510.749882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 2512.301585] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2512.871954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2517.335951] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 02:31:41 (1764747101) [ 2518.776829] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2522.316411] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2523.075128] Lustre: Failing over lustre-MDT0000 [ 2523.241838] Lustre: server umount lustre-MDT0000 complete [ 2526.176754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2526.179895] LustreError: Skipped 9 previous similar messages [ 2538.729062] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2538.730973] LDISKFS-fs (dm-0): recovery complete [ 2538.735170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.464826] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2544.122357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 2544.122444] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 2545.523587] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2546.202579] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2549.911715] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 02:32:14 (1764747134) [ 2550.297664] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2554.144391] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2554.801974] Lustre: Failing over lustre-MDT0000 [ 2554.891511] Lustre: server umount lustre-MDT0000 complete [ 2569.401643] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2569.403336] LDISKFS-fs (dm-0): recovery complete [ 2569.409394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.002617] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2574.843732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3169) [ 2574.843732] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 2576.309740] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2576.967457] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2581.001657] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 02:32:45 (1764747165) [ 2581.433499] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2586.008281] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2586.750472] Lustre: Failing over lustre-MDT0000 [ 2586.865560] Lustre: server umount lustre-MDT0000 complete [ 2602.458070] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2602.460238] LDISKFS-fs (dm-0): recovery complete [ 2602.465533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.573678] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2608.130268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 2608.130507] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3169) [ 2613.891261] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 02:33:17 (1764747197) [ 2615.430121] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2615.432133] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2615.434494] LustreError: 6489:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf532de450 x1850468880598784/t257698037777(0) o35->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:396/0 lens 392/456 e 0 to 0 dl 1764747211 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2616.765940] Lustre: Failing over lustre-MDT0000 [ 2616.914955] Lustre: server umount lustre-MDT0000 complete [ 2630.989195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.035948] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2636.286071] Lustre: 6489:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf7f47ad80 x1850468880598784/t257698037777(0) o35->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:417/0 lens 392/456 e 0 to 0 dl 1764747232 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2636.287356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3233) [ 2636.290854] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 2637.956066] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2638.771372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2642.811992] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 02:33:47 (1764747227) [ 2643.206165] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2643.208717] LustreError: 8071:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf78bb1180 x1850468880612096/t261993005072(0) o36->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:469/0 lens 504/448 e 0 to 0 dl 1764747284 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2647.374930] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2648.162932] Lustre: Failing over lustre-MDT0000 [ 2648.265018] Lustre: server umount lustre-MDT0000 complete [ 2663.119348] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2663.121141] LDISKFS-fs (dm-0): recovery complete [ 2663.127863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.937814] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2668.532057] Lustre: 14140:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ede45511f80 x1850468880612096/t261993005072(0) o36->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:495/0 lens 504/2880 e 0 to 0 dl 1764747310 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2668.538596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233) [ 2668.538608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 2669.939817] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2670.558943] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2674.567827] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 02:34:18 (1764747258) [ 2675.009351] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2675.011223] LustreError: 6487:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf7bd24000 x1850468880625920/t266287972368(0) o36->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:500/0 lens 504/448 e 0 to 0 dl 1764747315 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2676.399408] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2679.455171] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2680.215704] Lustre: Failing over lustre-MDT0000 [ 2680.321053] Lustre: server umount lustre-MDT0000 complete [ 2695.692243] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2695.694020] LDISKFS-fs (dm-0): recovery complete [ 2695.697304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2697.419113] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2701.301732] Lustre: 37131:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf44c7a300 x1850468880626688/t266287972369(0) o35->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:482/0 lens 392/456 e 0 to 0 dl 1764747297 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2701.304363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3265) [ 2701.304436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 2701.311698] Lustre: 37131:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 2706.449407] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 02:34:50 (1764747290) [ 2706.908295] LustreError: 6487:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf80e8f800 x1850468880639488/t270582939664(0) o36->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:532/0 lens 504/448 e 0 to 0 dl 1764747347 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2706.930957] LustreError: 6487:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 2708.292130] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2708.294235] Lustre: Skipped 1 previous similar message [ 2711.504942] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2712.262291] Lustre: Failing over lustre-MDT0000 [ 2712.372364] Lustre: server umount lustre-MDT0000 complete [ 2727.089996] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2727.091486] LDISKFS-fs (dm-0): recovery complete [ 2727.096589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.833866] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2732.527881] Lustre: 6487:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf6d7aca80 x1850468880639488/t270582939664(0) o36->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:558/0 lens 504/2880 e 0 to 0 dl 1764747373 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2732.534862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 2732.538500] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3297) [ 2736.961547] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 02:35:21 (1764747321) [ 2737.380703] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2738.719820] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2738.721633] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2738.723780] Lustre: Skipped 2 previous similar messages [ 2742.920409] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2743.777236] Lustre: Failing over lustre-MDT0000 [ 2743.886364] Lustre: server umount lustre-MDT0000 complete [ 2758.113433] LustreError: 10422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.119777] LustreError: 10422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 180 previous similar messages [ 2759.009206] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2759.010992] LDISKFS-fs (dm-0): recovery complete [ 2759.015535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.792240] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2764.274509] Lustre: 37131:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf7f3fce00 x1850468880652800/t274877906960(0) o35->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:545/0 lens 392/456 e 0 to 0 dl 1764747360 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2764.280629] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 2764.281204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3361) [ 2769.402254] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 02:35:53 (1764747353) [ 2769.798408] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 2769.800769] LustreError: 14140:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf7bd25500 x1850468880663936/t279172874255(0) o101->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:595/0 lens 664/608 e 0 to 0 dl 1764747410 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2769.809448] LustreError: 14140:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 2829.147498] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnecting [ 2829.151457] Lustre: Skipped 4 previous similar messages [ 2829.159942] Lustre: 6486:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf78bb1f80 x1850468880663936/t279172874255(0) o101->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:654/0 lens 664/3488 e 0 to 0 dl 1764747469 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2831.840023] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 02:36:56 (1764747416) [ 2834.979771] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2835.804699] Lustre: Failing over lustre-MDT0000 [ 2835.876104] LustreError: 138968:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2835.881674] LustreError: 138968:0:(obd_class.h:479:obd_check_dev()) Skipped 107 previous similar messages [ 2835.924470] Lustre: server umount lustre-MDT0000 complete [ 2835.937593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.942898] Lustre: Skipped 56 previous similar messages [ 2850.568374] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2850.570032] LDISKFS-fs (dm-0): recovery complete [ 2850.580902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.650964] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2850.654062] LustreError: Skipped 10 previous similar messages [ 2850.737077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.739798] Lustre: Skipped 13 previous similar messages [ 2850.775497] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2850.778523] Lustre: Skipped 15 previous similar messages [ 2852.052274] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2853.723380] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2853.727129] Lustre: Skipped 12 previous similar messages [ 2855.907415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2855.910081] Lustre: Skipped 55 previous similar messages [ 2855.919646] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2855.923411] Lustre: Skipped 12 previous similar messages [ 2855.939172] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 2855.939211] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3361) [ 2857.127417] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2857.673452] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2871.483985] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 02:37:35 (1764747455) [ 2874.525843] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2875.174173] Lustre: Failing over lustre-MDT0000 [ 2875.331778] Lustre: server umount lustre-MDT0000 complete [ 2889.466240] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2889.467883] LDISKFS-fs (dm-0): recovery complete [ 2889.470433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.847449] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2894.842524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 2894.842549] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3393) [ 2896.091414] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2896.676780] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2899.113403] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2904.676332] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 02:38:09 (1764747489) [ 2918.966313] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2919.686122] Lustre: Failing over lustre-MDT0000 [ 2920.039871] Lustre: server umount lustre-MDT0000 complete [ 2934.951176] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2934.952643] LDISKFS-fs (dm-0): recovery complete [ 2934.959563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.699250] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2940.510413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 2940.511779] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 2941.878952] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2942.518665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2962.489776] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 02:39:06 (1764747546) [ 2965.935887] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2966.656526] Lustre: Failing over lustre-MDT0000 [ 2966.884783] Lustre: server umount lustre-MDT0000 complete [ 2981.787428] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2981.789476] LDISKFS-fs (dm-0): recovery complete [ 2981.796453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.470905] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2987.037066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 2987.037080] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 2988.707688] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2989.410078] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2993.193268] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 2993.883947] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2996.777656] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 02:39:41 (1764747581) [ 3002.806515] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3063.524695] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3063.526577] Lustre: Skipped 1 previous similar message [ 3063.527807] LustreError: 6487:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf7bd25f80 x1850468883455488/t296352743435(0) o36->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:134/0 lens 66040/440 e 0 to 0 dl 1764747704 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3124.074081] Lustre: 6488:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf53289050 x1850468883455488/t296352743435(0) o36->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:194/0 lens 66040/440 e 0 to 0 dl 1764747764 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3127.224794] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3127.804890] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 02:41:52 (1764747712) [ 3132.062788] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3132.919578] Lustre: Failing over lustre-MDT0000 [ 3133.185298] Lustre: server umount lustre-MDT0000 complete [ 3135.457504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3135.460119] LustreError: Skipped 11 previous similar messages [ 3147.500130] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3147.502107] LDISKFS-fs (dm-0): recovery complete [ 3147.505429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3148.985637] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3153.081859] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 3153.081912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 3154.346530] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3154.924726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3158.683236] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 02:42:23 (1764747743) [ 3164.938439] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3168.288485] Lustre: Failing over lustre-OST0000 [ 3168.332614] Lustre: server umount lustre-OST0000 complete [ 3182.525067] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3182.527035] LDISKFS-fs (dm-2): recovery complete [ 3182.530310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3184.621861] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3196.172815] Lustre: Failing over lustre-OST0000 [ 3196.222443] Lustre: server umount lustre-OST0000 complete [ 3208.967760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3210.821430] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3213.791749] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3214.350233] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3248.243430] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 02:43:52 (1764747832) [ 3249.150748] Lustre: Failing over lustre-MDT0000 [ 3249.397957] Lustre: server umount lustre-MDT0000 complete [ 3262.332831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.776282] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3267.579425] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 3267.579626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 3275.058286] Lustre: Failing over lustre-MDT0000 [ 3275.204243] Lustre: server umount lustre-MDT0000 complete [ 3288.094849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3289.487432] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3293.685609] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 3293.685615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 3294.836453] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3295.360361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3298.712482] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 02:44:43 (1764747883) [ 3310.228030] Lustre: Failing over lustre-OST0000 [ 3310.262374] Lustre: server umount lustre-OST0000 complete [ 3323.107362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3325.063511] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3328.156145] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3328.673910] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3332.408222] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 02:45:16 (1764747916) [ 3333.043122] Lustre: Failing over lustre-MDT0000 [ 3333.346416] Lustre: server umount lustre-MDT0000 complete [ 3336.568629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3336.598802] Lustre: *** cfs_fail_loc=605, val=0*** [ 3336.599761] LustreError: 156753:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc08e8d90 failed: rc = -95 [ 3336.602090] LustreError: 156753:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 3336.603457] LustreError: 156753:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 3336.605121] LustreError: 156753:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 3336.606937] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3336.608852] LustreError: 156753:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3336.612861] Lustre: server umount lustre-MDT0000 complete [ 3336.614239] LustreError: 156753:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3339.329241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.805276] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3343.604962] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 02:45:28 (1764747928) [ 3344.887244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3344.887248] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3347.523600] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3348.659161] Lustre: Failing over lustre-MDT0000 [ 3348.881858] Lustre: server umount lustre-MDT0000 complete [ 3359.578857] LustreError: 17789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.585431] LustreError: 17789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 151 previous similar messages [ 3362.975225] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3362.976601] LDISKFS-fs (dm-0): recovery complete [ 3362.979348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3364.446172] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3364.700061] Lustre: *** cfs_fail_loc=707, val=0*** [ 3424.101329] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 3424.114274] Lustre: 158905:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3424.119113] Lustre: 158905:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 3424.207934] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 3424.208429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 3425.552459] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3426.061495] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3429.815249] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 02:46:54 (1764748014) [ 3453.610232] LustreError: 113785:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8edf6d4ff800 x1850468884334720/t0(0) o101->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:480/0 lens 664/0 e 0 to 0 dl 1764748050 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3453.617243] LustreError: 113785:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3464.640173] LustreError: 113785:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3464.647540] LustreError: 37131:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8edf7f7b1f80 x1850468884336000/t0(0) o35->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:491/0 lens 392/0 e 0 to 0 dl 1764748061 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3469.794027] LustreError: 8370:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8edf7bd27800 x1850468889647744/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:496/0 lens 544/0 e 0 to 0 dl 1764748066 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3469.801678] LustreError: 8370:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 48 previous similar messages [ 3476.370546] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 02:47:40 (1764748060) [ 3499.886566] LustreError: 8373:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3510.912165] LustreError: 8373:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3514.029471] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 02:48:18 (1764748098) [ 3537.239235] LustreError: 8071:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8edf76269880 x1850468884359808/t0(0) o101->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:563/0 lens 576/0 e 0 to 0 dl 1764748133 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3537.247144] LustreError: 8071:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3542.344153] LustreError: 8071:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3542.350174] LustreError: 8071:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8edf80ee7b80 x1850468884360704/t0(0) o101->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:568/0 lens 664/0 e 0 to 0 dl 1764748138 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3543.147324] LustreError: 6487:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 10000ms [ 3553.240122] LustreError: 6487:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3553.244935] LustreError: 6487:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8edf6cfa7100 x1850468884382464/t0(0) o101->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:618/0 lens 664/0 e 0 to 0 dl 1764748188 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3553.254695] LustreError: 6487:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 3565.107855] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 02:49:09 (1764748149) [ 3650.072524] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 02:50:34 (1764748234) [ 3673.072255] LustreError: 6488:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8edf6ca4df80 x1850468884439680/t0(0) o101->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:699/0 lens 576/0 e 0 to 0 dl 1764748269 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3673.078858] LustreError: 6488:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 3673.081267] LustreError: 6488:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3673.496099] LustreError: 6488:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3689.313798] LustreError: 6488:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3689.317554] LustreError: 6488:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3689.736099] LustreError: 6488:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3689.738702] LustreError: 6488:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3705.155306] LustreError: 6488:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8edf6d7ab480 x1850468884458880/t0(0) o36->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:731/0 lens 504/0 e 0 to 0 dl 1764748301 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 3705.161817] LustreError: 6488:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 82 previous similar messages [ 3717.250653] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 02:51:41 (1764748301) [ 3741.900525] Lustre: DEBUG MARKER: phase 2 [ 3745.025727] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 02:52:09 (1764748329) [ 3815.711920] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 02:53:20 (1764748400) [ 3816.201453] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 3816.750450] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 02:53:21 (1764748401) [ 3818.507845] Lustre: DEBUG MARKER: Started rundbench load pid=133529 ... [ 3822.252839] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3823.765027] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 3824.434194] Lustre: Failing over lustre-MDT0000 [ 3824.467361] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 3824.729907] LustreError: 165800:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3824.731774] LustreError: 165800:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages [ 3824.768902] Lustre: server umount lustre-MDT0000 complete [ 3828.704529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3828.706306] LustreError: Skipped 6 previous similar messages [ 3828.707332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3828.711548] Lustre: Skipped 39 previous similar messages [ 3839.052194] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3839.054311] LDISKFS-fs (dm-0): recovery complete [ 3839.058748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.093270] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3839.095895] LustreError: Skipped 8 previous similar messages [ 3839.170214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.172219] Lustre: Skipped 11 previous similar messages [ 3839.184217] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3839.187375] Lustre: Skipped 11 previous similar messages [ 3840.571021] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3840.858762] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3840.861247] Lustre: Skipped 11 previous similar messages [ 3844.579259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3844.580832] Lustre: Skipped 41 previous similar messages [ 3844.943433] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3844.946973] Lustre: Skipped 11 previous similar messages [ 3844.962578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5134 to 0x280000401:5153) [ 3844.962598] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5098 to 0x2c0000401:5121) [ 3846.140357] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3846.699866] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3851.754384] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3853.289381] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 3853.935636] Lustre: Failing over lustre-MDT0001 [ 3854.059382] Lustre: server umount lustre-MDT0001 complete [ 3868.486191] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 3868.488017] LDISKFS-fs (dm-1): recovery complete [ 3868.493140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3869.913632] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3873.780881] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:449) [ 3873.780881] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:449) [ 3874.955541] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3875.528451] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3880.568363] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3882.144913] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 3882.834449] Lustre: Failing over lustre-MDT0000 [ 3882.843989] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 3883.076516] Lustre: server umount lustre-MDT0000 complete [ 3897.073945] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3897.075816] LDISKFS-fs (dm-0): recovery complete [ 3897.078389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.478484] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3902.696761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5509 to 0x280000401:5537) [ 3902.696775] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5477 to 0x2c0000401:5505) [ 3903.969532] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3904.584655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3909.448604] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3910.961869] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 3911.619970] Lustre: Failing over lustre-MDT0001 [ 3911.820905] Lustre: server umount lustre-MDT0001 complete [ 3926.182127] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 3926.183616] LDISKFS-fs (dm-1): recovery complete [ 3926.187989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3927.641208] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3931.636919] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:481) [ 3931.636970] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:481) [ 3932.774260] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3933.403852] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3938.400718] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3939.929658] Lustre: DEBUG MARKER: test_70b fail mds1 5 times [ 3940.622885] Lustre: Failing over lustre-MDT0000 [ 3940.836099] Lustre: server umount lustre-MDT0000 complete [ 3954.970301] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3954.971410] LDISKFS-fs (dm-0): recovery complete [ 3954.973300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3956.436505] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3960.468402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5883 to 0x280000401:5921) [ 3960.468468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5851 to 0x2c0000401:5889) [ 3961.750851] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3962.415230] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3966.082349] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 02:55:50 (1764748550) [ 4090.893400] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4101.726628] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 4102.418103] Lustre: Failing over lustre-MDT0000 [ 4102.698600] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 4106.075091] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 4106.078702] Lustre: Skipped 4 previous similar messages [ 4107.868658] Lustre: server umount lustre-MDT0000 complete [ 4108.768700] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4108.773195] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 72 previous similar messages [ 4122.444217] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4122.445955] LDISKFS-fs (dm-0): recovery complete [ 4122.453529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4124.252717] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4130.377831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6866 to 0x280000401:6881) [ 4130.377834] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6834 to 0x2c0000401:6849) [ 4131.867889] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4132.589095] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4258.088083] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4269.139472] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 4269.999782] Lustre: Failing over lustre-MDT0001 [ 4270.061306] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4270.244252] Lustre: server umount lustre-MDT0001 complete [ 4285.813633] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4285.815344] LDISKFS-fs (dm-1): recovery complete [ 4285.823193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4287.636310] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4292.680937] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2154 to 0x2c0000400:2177) [ 4292.684147] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2155 to 0x280000400:2177) [ 4294.150940] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4294.769875] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4319.417656] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 03:01:43 (1764748903) [ 4443.197376] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4454.071499] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 4454.818787] Lustre: Failing over lustre-MDT0000 [ 4454.869302] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 4454.880919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.886078] Lustre: Skipped 24 previous similar messages [ 4454.888652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4454.891708] LustreError: Skipped 5 previous similar messages [ 4461.091897] LustreError: 178111:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4461.094763] LustreError: 178111:0:(obd_class.h:479:obd_check_dev()) Skipped 55 previous similar messages [ 4461.146181] Lustre: server umount lustre-MDT0000 complete [ 4476.333227] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4476.335195] LDISKFS-fs (dm-0): recovery complete [ 4476.344534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4476.432364] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4476.435957] LustreError: Skipped 3 previous similar messages [ 4476.554778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.558275] Lustre: Skipped 6 previous similar messages [ 4476.575881] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4476.579129] Lustre: Skipped 6 previous similar messages [ 4476.763671] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4476.768699] Lustre: Skipped 6 previous similar messages [ 4478.341478] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4482.020200] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4482.024096] Lustre: Skipped 24 previous similar messages [ 4483.402939] Lustre: lustre-MDT0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 4483.406297] Lustre: Skipped 6 previous similar messages [ 4483.423976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7836 to 0x280000401:7873) [ 4483.424454] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7804 to 0x2c0000401:7841) [ 4484.902908] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4485.593187] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4609.852759] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4620.600792] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 4621.237898] Lustre: Failing over lustre-MDT0000 [ 4621.792892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4621.794936] Lustre: Skipped 9 previous similar messages [ 4627.006966] Lustre: server umount lustre-MDT0000 complete [ 4641.257271] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4641.258656] LDISKFS-fs (dm-0): recovery complete [ 4641.261444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.800039] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4648.067741] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8205 to 0x2c0000401:8225) [ 4648.067746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8237 to 0x280000401:8257) [ 4649.454778] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4650.131719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4654.007800] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 03:07:18 (1764749238) [ 4777.180927] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4788.047211] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 4788.752186] Lustre: Failing over lustre-MDT0000 [ 4788.806465] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 4788.808825] Lustre: Skipped 5 previous similar messages [ 4788.963290] Lustre: server umount lustre-MDT0000 complete [ 4789.082969] LustreError: 17789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4789.090737] LustreError: 17789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 53 previous similar messages [ 4803.594224] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4803.595628] LDISKFS-fs (dm-0): recovery complete [ 4803.599975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4805.371917] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4812.986749] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9879 to 0x2c0000401:9921) [ 4812.987356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9911 to 0x280000401:9953) [ 4814.480067] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4815.177771] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4939.498139] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4950.332033] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 4951.001725] Lustre: Failing over lustre-MDT0000 [ 4951.039307] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 4951.362413] Lustre: server umount lustre-MDT0000 complete [ 4965.448412] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4965.450018] LDISKFS-fs (dm-0): recovery complete [ 4965.453212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.906430] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4973.904633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11564 to 0x280000401:11585) [ 4973.904662] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11531 to 0x2c0000401:11553) [ 4975.230307] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4975.921923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4979.900571] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 03:12:44 (1764749564) [ 4985.748789] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4987.240494] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 4987.872698] Lustre: Failing over lustre-OST0000 [ 4987.907478] Lustre: server umount lustre-OST0000 complete [ 5002.428933] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5002.430819] LDISKFS-fs (dm-2): recovery complete [ 5002.435304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5004.623818] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5008.337635] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5009.040627] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5018.216085] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5019.788849] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5020.448263] Lustre: Failing over lustre-OST0000 [ 5020.483508] Lustre: server umount lustre-OST0000 complete [ 5034.802990] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5034.804252] LDISKFS-fs (dm-2): recovery complete [ 5034.808108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5036.791636] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5040.201733] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5040.884686] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5046.500533] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 03:13:50 (1764749630) [ 5169.730065] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5172.434739] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5183.230559] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 5183.973391] Lustre: Failing over lustre-MDT0000 [ 5184.029358] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.48@tcp (stopping) [ 5184.134617] LustreError: 189536:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5184.136398] LustreError: 189536:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 5184.186525] Lustre: server umount lustre-MDT0000 complete [ 5185.466868] LustreError: 9446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749771 with bad export cookie 11785410073078518372 [ 5185.468154] Lustre: Failing over lustre-MDT0001 [ 5185.468609] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5185.468615] LustreError: Skipped 3 previous similar messages [ 5185.469888] LustreError: 9446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5185.714992] Lustre: server umount lustre-MDT0001 complete [ 5200.724615] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5200.725721] LDISKFS-fs (dm-1): recovery complete [ 5200.729475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5200.819571] LustreError: 190814:0:(llog.c:1643:llog_backup()) MGC192.168.202.148@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5200.826736] Lustre: 190814:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.148@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5200.851829] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5200.853512] LDISKFS-fs (dm-0): recovery complete [ 5200.856225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.864505] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.869778] Lustre: Skipped 22 previous similar messages [ 5204.960226] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764749774/real 1764749774] req@ffff8ede45510700 x1850468922937088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764749790 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5204.970256] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5207.008123] Lustre: 3627:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764749776/real 1764749776] req@ffff8edf7bf2f100 x1850468922937472/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764749792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5207.016287] Lustre: 3627:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5211.104594] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8edf7bf2ed80 x1850468922940800/t0(0) o250->MGC192.168.202.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5211.192932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5211.195883] Lustre: Skipped 5 previous similar messages [ 5211.205502] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5211.208077] Lustre: Skipped 5 previous similar messages [ 5211.273402] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 5211.276151] LustreError: Skipped 4 previous similar messages [ 5212.320091] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764749781/real 1764749781] req@ffff8edf7bf2c000 x1850468922938112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764749797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5212.332412] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5212.776389] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5212.891208] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5213.163281] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5213.166852] Lustre: Skipped 6 previous similar messages [ 5216.738071] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5216.740638] Lustre: Skipped 19 previous similar messages [ 5217.385029] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5217.388340] Lustre: Skipped 5 previous similar messages [ 5217.403971] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2847 to 0x280000400:2881) [ 5217.404130] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2848 to 0x2c0000400:2881) [ 5222.285902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11641 to 0x280000401:11681) [ 5222.285922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11609 to 0x2c0000401:11649) [ 5223.578365] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5224.178378] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5224.784194] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5228.096506] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 03:16:52 (1764749812) [ 5230.474753] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5231.299795] Lustre: Failing over lustre-MDT0000 [ 5231.408174] Lustre: server umount lustre-MDT0000 complete [ 5245.510323] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5245.511421] LDISKFS-fs (dm-0): recovery complete [ 5245.514473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.079574] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5248.860849] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5264.245730] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5264.310739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11683 to 0x280000401:11713) [ 5264.311021] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11609 to 0x2c0000401:11681) [ 5267.048817] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5268.222615] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5274.215397] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 03:17:38 (1764749858) [ 5277.973227] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5279.140199] Lustre: Failing over lustre-MDT0000 [ 5279.391574] Lustre: server umount lustre-MDT0000 complete [ 5293.784117] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5293.785833] LDISKFS-fs (dm-0): recovery complete [ 5293.789385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5294.948065] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5294.950101] LustreError: 190843:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8ede4e27f480 x1850468948105088/t352187318837(352187318837) o101->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:56/0 lens 520/664 e 0 to 0 dl 1764749891 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5295.244049] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5310.309543] Lustre: lustre-MDT0000: Client b9ec3cac-96a0-43fb-a74c-124f16871d0e (at 192.168.202.48@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5310.316350] Lustre: 191601:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ede47627480 x1850468948105088/t352187318837(352187318837) o101->b9ec3cac-96a0-43fb-a74c-124f16871d0e@192.168.202.48@tcp:71/0 lens 520/3488 e 0 to 0 dl 1764749906 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5310.350898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11609 to 0x2c0000401:11713) [ 5310.350898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:11745) [ 5311.851409] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5312.456110] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5316.082435] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 03:18:20 (1764749900) [ 5317.507422] Lustre: Failing over lustre-OST0000 [ 5317.549715] Lustre: server umount lustre-OST0000 complete [ 5318.853098] Lustre: Failing over lustre-MDT0000 [ 5318.948883] Lustre: server umount lustre-MDT0000 complete [ 5332.067326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.543483] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5334.340315] Lustre: lustre-MDT0000: Denying connection for new client eb7f1960-520e-4108-a9bf-8143a4a42395 (at 192.168.202.48@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5337.592273] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11609 to 0x2c0000401:11745) [ 5341.505231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5342.770856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:11777) [ 5343.478971] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5347.556199] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 03:18:52 (1764749932) [ 5347.932329] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5347.934249] LustreError: 173820:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8ede47958e00 x1850468924307200/t369367187463(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:109/0 lens 1312/4320 e 0 to 0 dl 1764749944 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5350.483466] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5351.124210] Lustre: Failing over lustre-MDT0000 [ 5351.409444] Lustre: server umount lustre-MDT0000 complete [ 5364.193099] Lustre: 190873:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764749933/real 1764749933] req@ffff8ede47958a80 x1850468924307200/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1764749949 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5365.650107] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5365.651888] LDISKFS-fs (dm-0): recovery complete [ 5365.655193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5367.042894] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5370.877103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:11809) [ 5370.877221] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:11777) [ 5372.136714] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5372.732033] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5376.358493] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 03:19:20 (1764749960) [ 5379.224097] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5381.756481] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5382.419150] Lustre: Failing over lustre-MDT0001 [ 5387.851604] Lustre: server umount lustre-MDT0001 complete [ 5390.683080] LustreError: 198444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5390.688394] LustreError: 198444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 132 previous similar messages [ 5402.277796] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5402.279214] LDISKFS-fs (dm-1): recovery complete [ 5402.283317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5403.844869] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5407.733300] Lustre: 190869:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ede4cd47480 x1850468954618368/t21474838701(0) o36->eb7f1960-520e-4108-a9bf-8143a4a42395@192.168.202.48@tcp:191/0 lens 560/2880 e 0 to 0 dl 1764750026 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5407.738413] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2892 to 0x280000400:2913) [ 5407.738430] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2892 to 0x2c0000400:2913) [ 5409.052439] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5409.699682] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5413.524923] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 03:19:57 (1764749997) [ 5413.885715] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5413.887575] Lustre: Skipped 1 previous similar message [ 5413.889495] LustreError: 8379:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf7faa1f80 x1850468924367232/t373662154785(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:175/0 lens 2520/4320 e 0 to 0 dl 1764750010 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5413.897450] LustreError: 8379:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 5416.494237] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5419.258897] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5420.003444] Lustre: Failing over lustre-MDT0000 [ 5420.253307] Lustre: server umount lustre-MDT0000 complete [ 5430.240184] Lustre: 201221:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764749999/real 1764749999] req@ffff8edf7faa1500 x1850468924367232/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1764750015 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5435.180506] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5435.182397] LDISKFS-fs (dm-0): recovery complete [ 5435.185960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5437.032298] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5440.593511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:11841) [ 5440.593799] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:11809) [ 5442.188791] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5442.929912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5445.193802] Lustre: Failing over lustre-MDT0001 [ 5445.303231] Lustre: server umount lustre-MDT0001 complete [ 5461.288810] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5461.291158] LDISKFS-fs (dm-1): recovery complete [ 5461.299790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5463.282348] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5466.624722] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2924 to 0x280000400:2945) [ 5466.624807] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2924 to 0x2c0000400:2945) [ 5468.359427] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5469.163574] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5474.487087] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 03:20:58 (1764750058) [ 5481.422185] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5485.062850] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5485.911879] Lustre: Failing over lustre-MDT0000 [ 5486.266029] Lustre: server umount lustre-MDT0000 complete [ 5487.984173] LustreError: 6472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750073 with bad export cookie 11785410073079466382 [ 5487.984879] Lustre: Failing over lustre-MDT0001 [ 5487.989941] LustreError: 6472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5487.995351] LustreError: 206523:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x20002a3e1:0xa:0x0].0x0 (ffff8ede4e386000) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5488.011295] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.48@tcp (stopping) [ 5488.013994] Lustre: Skipped 5 previous similar messages [ 5493.734498] LustreError: 206523:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8edf69bff480 x1850468924431360/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5493.744600] LustreError: 206523:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 5493.748295] LustreError: 206523:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5493.908482] Lustre: server umount lustre-MDT0001 complete [ 5510.567894] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5510.570892] LDISKFS-fs (dm-0): recovery complete [ 5510.575464] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5510.577884] LDISKFS-fs (dm-1): recovery complete [ 5510.586340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.595801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5510.743250] LustreError: 207620:0:(llog.c:1643:llog_backup()) MGC192.168.202.148@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5510.746633] Lustre: 207620:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.148@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5514.813237] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5515.046781] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5517.913349] Lustre: 207639:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf7faa3b80 x1850468954688000/t30064771142(0) o36->eb7f1960-520e-4108-a9bf-8143a4a42395@192.168.202.48@tcp:304/0 lens 560/2880 e 0 to 0 dl 1764750139 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5517.916771] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2956 to 0x2c0000400:2977) [ 5517.916946] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2956 to 0x280000400:2977) [ 5529.957987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:11873) [ 5529.958039] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:11841) [ 5532.189228] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5533.002651] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5533.865160] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5539.620937] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 03:22:03 (1764750123) [ 5540.163623] LustreError: 207639:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf789ec380 x1850468954721408/t34359738438(0) o36->eb7f1960-520e-4108-a9bf-8143a4a42395@192.168.202.48@tcp:340/0 lens 560/448 e 0 to 0 dl 1764750175 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5540.176390] LustreError: 207639:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 5546.784994] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5547.732147] Lustre: Failing over lustre-MDT0000 [ 5548.121730] Lustre: server umount lustre-MDT0000 complete [ 5563.995355] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5563.997124] LDISKFS-fs (dm-0): recovery complete [ 5564.007799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.261937] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5569.547677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:11905) [ 5569.548050] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:11873) [ 5571.248405] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5572.046864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5595.995427] Lustre: lustre-MDT0001: Client eb7f1960-520e-4108-a9bf-8143a4a42395 (at 192.168.202.48@tcp) reconnecting [ 5596.000226] Lustre: Skipped 2 previous similar messages [ 5596.014635] Lustre: 207638:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf7cb7e680 x1850468954721408/t34359738438(0) o36->eb7f1960-520e-4108-a9bf-8143a4a42395@192.168.202.48@tcp:396/0 lens 560/2880 e 0 to 0 dl 1764750231 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5600.714690] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 03:23:04 (1764750184) [ 5601.413778] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5601.417964] Lustre: Skipped 2 previous similar messages [ 5605.747890] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5606.772387] Lustre: Failing over lustre-MDT0001 [ 5606.904374] Lustre: server umount lustre-MDT0001 complete [ 5623.573063] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5623.575465] LDISKFS-fs (dm-1): recovery complete [ 5623.588365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5626.045564] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5628.917543] Lustre: 209444:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf58fc9180 x1850468954756480/t34359738509(0) o36->eb7f1960-520e-4108-a9bf-8143a4a42395@192.168.202.48@tcp:390/0 lens 560/2880 e 0 to 0 dl 1764750225 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5628.921380] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2998 to 0x280000400:3041) [ 5628.921789] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2998 to 0x2c0000400:3041) [ 5630.523839] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5631.349803] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5636.730931] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 03:23:40 (1764750220) [ 5643.943356] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5647.740448] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5648.825221] Lustre: Failing over lustre-MDT0000 [ 5649.102482] Lustre: server umount lustre-MDT0000 complete [ 5664.737827] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764750234/real 1764750234] req@ffff8edf44409500 x1850468924555008/t0(0) o400->MGC192.168.202.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764750250 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5665.756662] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5665.758644] LDISKFS-fs (dm-0): recovery complete [ 5665.765798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.978076] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8edf4457c700 x1850468924563328/t0(0) o250->MGC192.168.202.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5676.942312] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5680.142103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:11937) [ 5680.142124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:11905) [ 5681.806802] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5682.652377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5684.943325] Lustre: Failing over lustre-MDT0001 [ 5685.039820] Lustre: server umount lustre-MDT0001 complete [ 5701.047212] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5701.049176] LDISKFS-fs (dm-1): recovery complete [ 5701.058044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5703.088508] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5706.240496] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3052 to 0x2c0000400:3073) [ 5706.240606] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3052 to 0x280000400:3073) [ 5708.005969] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5708.734809] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5713.549555] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 03:24:57 (1764750297) [ 5714.040685] LustreError: 207638:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf58fcb100 x1850468954827136/t42949673030(0) o36->eb7f1960-520e-4108-a9bf-8143a4a42395@192.168.202.48@tcp:475/0 lens 560/448 e 0 to 0 dl 1764750310 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5714.049368] LustreError: 207638:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 2 previous similar messages [ 5720.526917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5724.569258] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5725.484751] Lustre: Failing over lustre-MDT0000 [ 5725.830524] Lustre: server umount lustre-MDT0000 complete [ 5727.589836] LustreError: 9446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750313 with bad export cookie 11785410073079480165 [ 5727.590348] Lustre: Failing over lustre-MDT0001 [ 5727.595316] LustreError: 9446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5727.725263] Lustre: server umount lustre-MDT0001 complete [ 5744.215487] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5744.217360] LDISKFS-fs (dm-0): recovery complete [ 5744.227500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.269805] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5744.276800] LDISKFS-fs (dm-1): recovery complete [ 5744.293623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5748.192177] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764750317/real 1764750317] req@ffff8edf7625ed80 x1850468924614016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764750333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5748.207118] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5753.313795] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8edf7625c700 x1850468924616192/t0(0) o250->MGC192.168.202.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5755.394830] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5755.596183] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5758.666345] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3084 to 0x2c0000400:3105) [ 5758.666356] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3084 to 0x280000400:3105) [ 5758.671544] Lustre: 218271:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ede45510380 x1850468954827136/t42949673030(0) o36->eb7f1960-520e-4108-a9bf-8143a4a42395@192.168.202.48@tcp:520/0 lens 560/2880 e 0 to 0 dl 1764750355 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5758.671555] Lustre: 218271:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5758.705165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:11969) [ 5758.705786] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:11937) [ 5760.565580] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5761.501649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5762.379275] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5768.028592] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 03:25:52 (1764750352) [ 5772.446918] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5773.414032] Lustre: Failing over lustre-MDT0001 [ 5773.422107] LustreError: 219934:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x20002a3e1:0x17:0x0].0xf7117594 (ffff8edf70caa900) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5773.431161] LustreError: 219934:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message [ 5779.055036] Lustre: server umount lustre-MDT0001 complete [ 5794.930514] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5794.932552] LDISKFS-fs (dm-1): recovery complete [ 5794.942255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5796.799304] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5800.450912] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3116 to 0x280000400:3137) [ 5800.451171] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3116 to 0x2c0000400:3137) [ 5802.163188] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5802.887355] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5807.209266] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 03:26:31 (1764750391) [ 5811.297248] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5812.140360] Lustre: Failing over lustre-MDT0000 [ 5812.283319] LustreError: 221789:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5812.286217] LustreError: 221789:0:(obd_class.h:479:obd_check_dev()) Skipped 147 previous similar messages [ 5812.335541] Lustre: server umount lustre-MDT0000 complete [ 5815.777183] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5815.778849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5815.784183] Lustre: Skipped 58 previous similar messages [ 5815.786748] LustreError: Skipped 14 previous similar messages [ 5823.968230] Lustre: 220562:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764750393/real 1764750393] req@ffff8ede47680e00 x1850468924680192/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1744/4320 e 0 to 1 dl 1764750409 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5823.994991] Lustre: 220562:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 5827.801729] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5827.805513] LDISKFS-fs (dm-0): recovery complete [ 5827.811449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5827.873105] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5827.876683] LustreError: Skipped 9 previous similar messages [ 5827.974590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5827.977720] Lustre: Skipped 18 previous similar messages [ 5827.996573] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5827.999275] Lustre: Skipped 18 previous similar messages [ 5829.787530] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5831.004337] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5831.008798] Lustre: Skipped 17 previous similar messages [ 5833.186304] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5833.189233] Lustre: Skipped 61 previous similar messages [ 5833.217112] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5833.223865] Lustre: Skipped 18 previous similar messages [ 5833.240612] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:11969) [ 5833.241234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:12001) [ 5835.015355] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5835.772716] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5840.477156] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 03:27:04 (1764750424) [ 5845.314131] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5849.314296] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5850.421589] Lustre: Failing over lustre-MDT0000 [ 5850.722800] Lustre: server umount lustre-MDT0000 complete [ 5866.987132] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5866.988998] LDISKFS-fs (dm-0): recovery complete [ 5867.001182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.272232] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5872.679271] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:12001) [ 5872.688680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:12033) [ 5874.504560] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5875.291942] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5877.659201] Lustre: Failing over lustre-MDT0001 [ 5877.796255] Lustre: server umount lustre-MDT0001 complete [ 5893.977116] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5893.978661] LDISKFS-fs (dm-1): recovery complete [ 5893.987389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5896.317428] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5899.266191] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3116 to 0x280000400:3169) [ 5899.266221] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3116 to 0x2c0000400:3169) [ 5900.896758] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5901.716483] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5906.018259] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 03:28:10 (1764750490) [ 5906.561875] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5906.564426] Lustre: Skipped 5 previous similar messages [ 5910.121210] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5913.420904] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5914.202824] Lustre: Failing over lustre-MDT0000 [ 5914.447001] Lustre: server umount lustre-MDT0000 complete [ 5915.975731] LustreError: 6472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750501 with bad export cookie 11785410073079490546 [ 5915.977378] Lustre: Failing over lustre-MDT0001 [ 5915.980483] LustreError: 6472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5915.982276] LustreError: 179333:0:(tgt_grant.c:234:tgt_grant_sanity_check()) mdt_destroy_export: tot_granted 0 != fo_tot_granted 2097152 [ 5915.983638] LustreError: 227735:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x20002a3e1:0x1f:0x0].0x0 (ffff8edf42f44900) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5915.986605] LustreError: 227735:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8edf48617100 x1850468924763136/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5916.004498] LustreError: 227735:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5916.127153] Lustre: server umount lustre-MDT0001 complete [ 5931.732527] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5931.736109] LDISKFS-fs (dm-0): recovery complete [ 5931.737266] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5931.740947] LDISKFS-fs (dm-1): recovery complete [ 5931.743586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5931.753319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5931.866925] LustreError: 228836:0:(llog.c:1643:llog_backup()) MGC192.168.202.148@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5931.870160] Lustre: 228836:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.202.148@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5943.210253] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5943.350909] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5947.668928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:12033) [ 5947.668933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:12065) [ 5951.792125] Lustre: 228846:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8edf6d8d8e00 x1850468954906496/t55834574850(0) o36->eb7f1960-520e-4108-a9bf-8143a4a42395@192.168.202.48@tcp:738/0 lens 496/2888 e 0 to 0 dl 1764750573 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 5951.797967] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3116 to 0x2c0000400:3201) [ 5951.798063] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3116 to 0x280000400:3201) [ 5951.801526] Lustre: 228846:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5953.447850] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5954.183795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5954.915682] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5959.367449] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 03:29:03 (1764750543) [ 5964.255239] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5965.193592] Lustre: Failing over lustre-MDT0000 [ 5965.397374] Lustre: server umount lustre-MDT0000 complete [ 5974.496160] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764750505/real 1764750505] req@ffff8edf4865ca80 x1850468924763648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764750560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5974.506155] Lustre: 3626:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5975.389697] Lustre: lustre-MDT0001: Client eb7f1960-520e-4108-a9bf-8143a4a42395 (at 192.168.202.48@tcp) reconnecting [ 5975.393719] Lustre: Skipped 1 previous similar message [ 5980.629771] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5980.631690] LDISKFS-fs (dm-0): recovery complete [ 5980.636966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.331902] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5986.310678] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:12065) [ 5986.310706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:12097) [ 5987.824312] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5988.460461] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5992.554725] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 03:29:36 (1764750576) [ 5993.002953] LustreError: 229599:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8edf48fe4700 x1850468954934400/t60129542149(0) o36->eb7f1960-520e-4108-a9bf-8143a4a42395@192.168.202.48@tcp:754/0 lens 496/456 e 0 to 0 dl 1764750589 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 5993.012145] LustreError: 229599:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 5 previous similar messages [ 5995.854891] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5996.563586] Lustre: Failing over lustre-MDT0001 [ 5996.675058] Lustre: server umount lustre-MDT0001 complete [ 5998.945263] LustreError: 228869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5998.953985] LustreError: 228869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 225 previous similar messages [ 6011.731193] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6011.732928] LDISKFS-fs (dm-1): recovery complete [ 6011.738757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6013.372550] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6017.028447] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3116 to 0x280000400:3233) [ 6017.028650] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3116 to 0x2c0000400:3233) [ 6018.580210] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6019.265917] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6023.463057] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 03:30:07 (1764750607) [ 6027.000125] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6030.210686] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6030.982321] Lustre: Failing over lustre-MDT0000 [ 6031.212071] Lustre: server umount lustre-MDT0000 complete [ 6046.246489] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6046.248427] LDISKFS-fs (dm-0): recovery complete [ 6046.254789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.999425] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6051.852306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:12129) [ 6051.852321] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:12097) [ 6053.458969] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6054.170392] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6056.386419] Lustre: Failing over lustre-MDT0001 [ 6056.510164] Lustre: server umount lustre-MDT0001 complete [ 6071.554302] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6071.556212] LDISKFS-fs (dm-1): recovery complete [ 6071.562922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6073.298048] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6076.920749] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3116 to 0x2c0000400:3265) [ 6076.921561] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3116 to 0x280000400:3265) [ 6078.530254] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6079.272144] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6083.372571] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 03:31:07 (1764750667) [ 6086.850855] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6089.786728] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6090.567518] Lustre: Failing over lustre-MDT0000 [ 6090.758578] Lustre: server umount lustre-MDT0000 complete [ 6092.287095] LustreError: 144614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750677 with bad export cookie 11785410073079498078 [ 6092.287673] Lustre: Failing over lustre-MDT0001 [ 6092.293332] LustreError: 144614:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6092.418980] Lustre: server umount lustre-MDT0001 complete [ 6107.750612] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6107.753913] LDISKFS-fs (dm-0): recovery complete [ 6107.759491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.761397] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6107.765330] LDISKFS-fs (dm-1): recovery complete [ 6107.771436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6112.736307] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764750682/real 1764750682] req@ffff8ede45510380 x1850468924899072/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764750698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6112.745512] Lustre: 3628:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 6116.832496] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8edf473f1500 x1850468924901248/t0(0) o250->MGC192.168.202.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6118.593693] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6118.750887] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6123.135040] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:12129) [ 6123.135039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11715 to 0x280000401:12161) [ 6124.202911] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3116 to 0x280000400:3297) [ 6124.202916] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3116 to 0x2c0000400:3297) [ 6125.349447] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6125.956704] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6126.526257] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6130.258343] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 03:31:54 (1764750714) [ 6130.941595] Lustre: 240693:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting eb7f1960-520e-4108-a9bf-8143a4a42395 at adminstrative request [ 6135.877258] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 03:32:00 (1764750720) [ 6137.951088] Lustre: Failing over lustre-MDT0000 [ 6138.095880] Lustre: server umount lustre-MDT0000 complete [ 6151.069318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6152.570432] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6156.300026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12212 to 0x280000401:12257) [ 6156.300052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12181 to 0x2c0000401:12225) [ 6157.483925] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6158.097646] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6161.959068] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 03:32:26 (1764750746) [ 6166.762809] Lustre: Failing over lustre-OST0000 [ 6166.833957] Lustre: server umount lustre-OST0000 complete [ 6180.153319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6182.486396] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6185.950908] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6186.585540] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6190.616252] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 03:32:54 (1764750774) [ 6192.075225] Lustre: Failing over lustre-MDT0000 [ 6192.096872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6192.099566] Lustre: Skipped 17 previous similar messages [ 6192.267641] Lustre: server umount lustre-MDT0000 complete [ 6195.082399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6196.702154] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6197.498592] Lustre: lustre-MDT0000: Denying connection for new client 03b7ab51-8c9c-4062-8960-15abc3f53365 (at 192.168.202.48@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 6197.506103] Lustre: Skipped 2 previous similar messages [ 6200.313125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12212 to 0x280000401:12289) [ 6200.313132] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12181 to 0x2c0000401:12257) [ 6205.315624] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 03:33:09 (1764750789) [ 6208.606219] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6209.643805] Lustre: Failing over lustre-OST0000 [ 6209.683721] Lustre: server umount lustre-OST0000 complete [ 6224.581784] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6224.584968] LDISKFS-fs (dm-2): recovery complete [ 6224.589551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6226.841464] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6230.682496] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6231.381082] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6235.645352] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 03:33:39 (1764750819) [ 6239.133910] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6241.130071] Lustre: Failing over lustre-OST0000 [ 6241.168593] Lustre: server umount lustre-OST0000 complete [ 6255.800325] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6255.801986] LDISKFS-fs (dm-2): recovery complete [ 6255.806848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6257.802231] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.48@tcp inode [0x200031141:0x5:0x0] object 0x280000401:12291 extent [0-4194303]: client csum a9ec9d31, server csum 6ebbfa8f [ 6258.054972] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6261.323624] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6261.940261] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6265.720791] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 03:34:10 (1764750850) [ 6268.543107] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6271.369465] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6273.844147] Lustre: Failing over lustre-MDT0000 [ 6274.120679] Lustre: server umount lustre-MDT0000 complete [ 6275.506381] LustreError: 9446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750861 with bad export cookie 11785410073079524867 [ 6275.509637] Lustre: Failing over lustre-OST0000 [ 6275.510786] LustreError: 9446:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6275.564507] Lustre: server umount lustre-OST0000 complete [ 6290.651578] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6290.653428] LDISKFS-fs (dm-0): recovery complete [ 6290.657887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6301.152950] LustreError: 3625:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ede44730000 x1850468925047936/t0(0) o250->MGC192.168.202.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6302.886056] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6306.415031] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12181 to 0x2c0000401:12289) [ 6318.534185] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6318.536090] LDISKFS-fs (dm-2): recovery complete [ 6318.541396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6321.007483] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6327.822312] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 03:35:12 (1764750912) [ 6333.243811] Lustre: Failing over lustre-OST0000 [ 6333.314439] Lustre: server umount lustre-OST0000 complete [ 6334.750552] Lustre: Failing over lustre-MDT0000 [ 6335.095896] Lustre: server umount lustre-MDT0000 complete [ 6348.369018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6350.104419] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6353.918802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12181 to 0x2c0000401:12321) [ 6357.561497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6359.707072] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6360.493771] Lustre: lustre-OST0000: Denying connection for new client 2e39cc73-979c-4740-8fad-f581dac33261 (at 192.168.202.48@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 6360.499410] Lustre: Skipped 1 previous similar message [ 6365.531363] Lustre: lustre-OST0000: Denying connection for new client 2e39cc73-979c-4740-8fad-f581dac33261 (at 192.168.202.48@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 6365.537276] Lustre: Skipped 1 previous similar message [ 6375.770861] Lustre: lustre-OST0000: Denying connection for new client 2e39cc73-979c-4740-8fad-f581dac33261 (at 192.168.202.48@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 6375.775444] Lustre: Skipped 1 previous similar message [ 6396.250848] Lustre: lustre-OST0000: Denying connection for new client 2e39cc73-979c-4740-8fad-f581dac33261 (at 192.168.202.48@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:33 [ 6396.257400] Lustre: Skipped 3 previous similar messages [ 6429.500281] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6429.502801] Lustre: 253739:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 03b7ab51-8c9c-4062-8960-15abc3f53365@ [ 6429.507357] Lustre: 253739:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6429.510880] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6429.521749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12340 to 0x280000401:12361) [ 6433.557293] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 6438.476062] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 6441.122261] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 03:37:05 (1764751025) [ 6442.722947] Lustre: Failing over lustre-OST0000 [ 6442.744067] LustreError: 254977:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6442.746382] LustreError: 254977:0:(obd_class.h:479:obd_check_dev()) Skipped 139 previous similar messages [ 6442.761902] Lustre: server umount lustre-OST0000 complete [ 6445.024707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6445.028072] LustreError: Skipped 16 previous similar messages [ 6445.029740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.034148] Lustre: Skipped 57 previous similar messages [ 6456.266037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6456.329051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6456.333229] Lustre: Skipped 19 previous similar messages [ 6456.338549] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6456.341834] Lustre: Skipped 19 previous similar messages [ 6457.691327] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6457.696690] Lustre: Skipped 19 previous similar messages [ 6457.842550] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6457.842597] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6457.845632] Lustre: Skipped 19 previous similar messages [ 6457.849156] Lustre: Skipped 60 previous similar messages [ 6458.639156] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6463.422133] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 03:37:27 (1764751047) [ 6465.071057] Lustre: Failing over lustre-OST0000 [ 6465.118630] Lustre: server umount lustre-OST0000 complete [ 6478.358492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6480.046612] LustreError: 256722:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6480.050590] LustreError: 256722:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 78 previous similar messages [ 6480.600192] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6485.984175] Lustre: *** cfs_fail_loc=715, val=40*** [ 6495.578875] Lustre: lustre-OST0000: Client 2e39cc73-979c-4740-8fad-f581dac33261 (at 192.168.202.48@tcp) reconnected, waiting for 3 clients in recovery for 0:53 [ 6496.224152] Lustre: 3625:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764751065/real 1764751065] req@ffff8edf7625f100 x1850468925158272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764751081 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 6496.235204] Lustre: 3625:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 6496.238847] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:53 [ 6496.241614] Lustre: Skipped 1 previous similar message [ 6501.856263] Lustre: *** cfs_fail_loc=715, val=40*** [ 6501.857940] Lustre: Skipped 2 previous similar messages [ 6511.584678] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:37 [ 6511.587411] Lustre: Skipped 1 previous similar message [ 6517.728325] Lustre: *** cfs_fail_loc=715, val=40*** [ 6517.731117] Lustre: Skipped 2 previous similar messages [ 6520.096742] LustreError: 256722:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6520.101503] LustreError: 256722:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 78 previous similar messages [ 6521.867830] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6522.580504] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6526.736421] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 03:38:31 (1764751111) [ 6528.613073] Lustre: Failing over lustre-MDT0000 [ 6528.959469] Lustre: server umount lustre-MDT0000 complete [ 6542.354526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6542.403809] LustreError: MGC192.168.202.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6542.409303] LustreError: Skipped 9 previous similar messages [ 6544.153361] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6547.947076] LustreError: 258165:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6548.960269] Lustre: *** cfs_fail_loc=715, val=80*** [ 6548.962043] Lustre: Skipped 2 previous similar messages [ 6554.080203] Lustre: *** cfs_fail_loc=715, val=80*** [ 6559.066941] Lustre: lustre-MDT0000: Client 2e39cc73-979c-4740-8fad-f581dac33261 (at 192.168.202.48@tcp) reconnected, waiting for 2 clients in recovery for 1:29 [ 6559.071705] Lustre: Skipped 1 previous similar message [ 6564.320612] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6565.344195] Lustre: *** cfs_fail_loc=715, val=80*** [ 6574.426711] Lustre: lustre-MDT0000: Client 2e39cc73-979c-4740-8fad-f581dac33261 (at 192.168.202.48@tcp) reconnected, waiting for 2 clients in recovery for 1:14 [ 6579.680723] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6585.824253] Lustre: *** cfs_fail_loc=715, val=80*** [ 6585.825911] Lustre: Skipped 2 previous similar messages [ 6590.811864] Lustre: lustre-MDT0000: Client 2e39cc73-979c-4740-8fad-f581dac33261 (at 192.168.202.48@tcp) reconnected, waiting for 2 clients in recovery for 0:57 [ 6596.064517] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6612.452877] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6618.592187] Lustre: *** cfs_fail_loc=715, val=80*** [ 6618.593905] Lustre: Skipped 3 previous similar messages [ 6622.554967] Lustre: lustre-MDT0000: Client 2e39cc73-979c-4740-8fad-f581dac33261 (at 192.168.202.48@tcp) reconnected, waiting for 2 clients in recovery for 0:25 [ 6622.559685] Lustre: Skipped 1 previous similar message [ 6628.032263] LustreError: 258165:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6628.043855] Lustre: 258165:0:(ldlm_lib.c:2931:target_recovery_thread()) too long recovery - read logs [ 6628.053451] LustreError: dumping log to /tmp/lustre-log.1764751213.258165 [ 6628.130945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12375 to 0x280000401:12393) [ 6628.131319] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12334 to 0x2c0000401:12353) [ 6629.713102] Lustre: DEBUG MARKER: oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6630.421381] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6634.775663] Lustre: DEBUG MARKER: == replay-single test complete, duration 6486 sec ======== 03:40:19 (1764751219) [ 6635.475275] Lustre: DEBUG MARKER: === replay-single: start cleanup 03:40:19 (1764751219) === [ 6639.535253] Lustre: DEBUG MARKER: === replay-single: finish cleanup 03:40:23 (1764751223) === [ 6675.938421] Lustre: server umount lustre-MDT0000 complete [ 6678.857631] LustreError: 6470:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764751264 with bad export cookie 11785410073079537600 [ 6678.861712] LustreError: 6470:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6678.958353] Lustre: server umount lustre-MDT0001 complete [ 6692.363966] Lustre: server umount lustre-OST0000 complete [ 6695.552256] Lustre: server umount lustre-OST0001 complete [ 6701.740677] Lustre: DEBUG MARKER: oleg248-server.virtnet: executing unload_modules_local [ 6702.904095] Key type lgssc unregistered [ 6703.057422] LNet: 261322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6703.061152] LNetError: 261322:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6703.072338] LNet: Removed LNI 192.168.202.148@tcp [ 6703.471332] Key type .llcrypt unregistered [ 6703.472634] Key type ._llcrypt unregistered