[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1959752987 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.002007] APIC: Switch to symmetric I/O mode setup [ 0.005098] x2apic enabled [ 0.007006] Switched APIC routing to physical x2apic. [ 0.008137] kvm-guest: setup PV IPIs [ 0.015000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.015000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.015028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.016014] pid_max: default: 32768 minimum: 301 [ 0.017162] LSM: Security Framework initializing [ 0.018064] Yama: becoming mindful. [ 0.019073] SELinux: Initializing. [ 0.021009] *** VALIDATE selinux *** [ 0.030288] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.036709] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.037160] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.038135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.039121] *** VALIDATE tmpfs *** [ 0.040506] *** VALIDATE proc *** [ 0.041297] *** VALIDATE cgroup *** [ 0.042018] *** VALIDATE cgroup2 *** [ 0.044092] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.045209] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.046025] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.048055] Spectre V2 : User space: Vulnerable [ 0.049016] Speculative Store Bypass: Vulnerable [ 0.053643] debug: unmapping init [mem 0xffffffffbae59000-0xffffffffbae60fff] [ 0.056704] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.057717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.058031] ... version: 2 [ 0.059023] ... bit width: 48 [ 0.060020] ... generic registers: 4 [ 0.061024] ... value mask: 0000ffffffffffff [ 0.062020] ... max period: 00007fffffffffff [ 0.063019] ... fixed-purpose events: 3 [ 0.064044] ... event mask: 000000070000000f [ 0.066521] rcu: Hierarchical SRCU implementation. [ 0.070669] smp: Bringing up secondary CPUs ... [ 0.072463] x86: Booting SMP configuration: [ 0.073046] .... node #0, CPUs: #1 #2 #3 [ 0.082073] smp: Brought up 1 node, 4 CPUs [ 0.084050] smpboot: Max logical packages: 1 [ 0.085019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.165461] node 0 deferred pages initialised in 78ms [ 0.176008] devtmpfs: initialized [ 0.178250] x86/mm: Memory block size: 128MB [ 0.197471] gcov: version magic: 0x41383552 [ 0.205676] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.209736] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.218118] pinctrl core: initialized pinctrl subsystem [ 0.225512] [ 0.226012] ************************************************************* [ 0.235017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.242014] ** ** [ 0.247012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255014] ** ** [ 0.257019] ** This means that this kernel is built to expose internal ** [ 0.264018] ** IOMMU data structures, which may compromise security on ** [ 0.274017] ** your system. ** [ 0.285025] ** ** [ 0.289017] ** If you see this message and you are not debugging the ** [ 0.300023] ** kernel, report this immediately to your vendor! ** [ 0.305016] ** ** [ 0.306017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.311018] ************************************************************* [ 0.322516] NET: Registered protocol family 16 [ 0.327380] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.336096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.342065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.353204] cpuidle: using governor menu [ 0.357040] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.363104] PCI: Using configuration type 1 for base access [ 0.369140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.394098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.401022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.410034] cryptd: max_cpu_qlen set to 1000 [ 0.412387] ACPI: Added _OSI(Module Device) [ 0.414012] ACPI: Added _OSI(Processor Device) [ 0.416011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.417115] ACPI: Added _OSI(Processor Aggregator Device) [ 0.426771] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.455127] ACPI: Interpreter enabled [ 0.461507] ACPI: PM: (supports S0 S3 S4 S5) [ 0.465017] ACPI: Using IOAPIC for interrupt routing [ 0.469104] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.481519] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.502257] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.511046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.514019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.518081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.524743] acpiphp: Slot [2] registered [ 0.526125] acpiphp: Slot [5] registered [ 0.528120] acpiphp: Slot [6] registered [ 0.530144] acpiphp: Slot [7] registered [ 0.531122] acpiphp: Slot [8] registered [ 0.532284] acpiphp: Slot [9] registered [ 0.534120] acpiphp: Slot [10] registered [ 0.536133] acpiphp: Slot [3] registered [ 0.538103] acpiphp: Slot [4] registered [ 0.540126] acpiphp: Slot [11] registered [ 0.542114] acpiphp: Slot [12] registered [ 0.543111] acpiphp: Slot [13] registered [ 0.545128] acpiphp: Slot [14] registered [ 0.547114] acpiphp: Slot [15] registered [ 0.549229] acpiphp: Slot [16] registered [ 0.550222] acpiphp: Slot [17] registered [ 0.551290] acpiphp: Slot [18] registered [ 0.553096] acpiphp: Slot [19] registered [ 0.555106] acpiphp: Slot [20] registered [ 0.557092] acpiphp: Slot [21] registered [ 0.559090] acpiphp: Slot [22] registered [ 0.560172] acpiphp: Slot [23] registered [ 0.562179] acpiphp: Slot [24] registered [ 0.564251] acpiphp: Slot [25] registered [ 0.565099] acpiphp: Slot [26] registered [ 0.567099] acpiphp: Slot [27] registered [ 0.569085] acpiphp: Slot [28] registered [ 0.570086] acpiphp: Slot [29] registered [ 0.572125] acpiphp: Slot [30] registered [ 0.574109] acpiphp: Slot [31] registered [ 0.576063] PCI host bridge to bus 0000:00 [ 0.577021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.580022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.583020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.586022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.589022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.592025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.594168] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.598329] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.602561] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.613016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.618061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.623021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.626018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.629018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.632815] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.635864] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.638044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.641010] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.647015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.661014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.668018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.675670] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.687017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.734029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.764029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 1.044000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.079016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.090018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.111016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 1.121033] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.128018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.135021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.152018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.165082] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.172019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.179015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.195019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.206117] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.214014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.222018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.247019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.260000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.280021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.300016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.328021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.344434] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.346596] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.349667] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.352520] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.354274] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.359342] iommu: Default domain type: Passthrough [ 1.360691] SCSI subsystem initialized [ 1.361231] ACPI: bus type USB registered [ 1.362150] usbcore: registered new interface driver usbfs [ 1.363118] usbcore: registered new interface driver hub [ 1.365166] usbcore: registered new device driver usb [ 1.368131] pps_core: LinuxPPS API ver. 1 registered [ 1.370016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.374082] PTP clock support registered [ 1.376038] EDAC MC: Ver: 3.0.0 [ 1.378175] PCI: Using ACPI for IRQ routing [ 1.380117] NetLabel: Initializing [ 1.381010] NetLabel: domain hash size = 128 [ 1.383018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.386155] NetLabel: unlabeled traffic allowed by default [ 1.389256] vgaarb: loaded [ 1.391314] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.395007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.478129] clocksource: Switched to clocksource kvm-clock [ 1.789603] hrtimer: interrupt took 8772636 ns [ 2.850628] VFS: Disk quotas dquot_6.6.0 [ 2.858115] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.875462] *** VALIDATE ramfs *** [ 2.888680] *** VALIDATE hugetlbfs *** [ 2.900492] pnp: PnP ACPI init [ 2.919143] pnp: PnP ACPI: found 6 devices [ 3.033109] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 3.037439] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 3.039589] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 3.041898] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 3.044513] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 3.047571] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 3.050615] NET: Registered protocol family 2 [ 3.053398] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 3.059097] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 3.062918] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 3.068626] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 3.072308] TCP: Hash tables configured (established 65536 bind 65536) [ 3.075304] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 3.078380] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 3.081454] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 3.084726] NET: Registered protocol family 1 [ 3.088269] RPC: Registered named UNIX socket transport module. [ 3.090854] RPC: Registered udp transport module. [ 3.092438] RPC: Registered tcp transport module. [ 3.094293] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 3.096876] NET: Registered protocol family 44 [ 3.098324] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 3.100661] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 3.103043] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 3.105386] PCI: CLS 0 bytes, default 64 [ 3.106896] Unpacking initramfs... [ 12.311311] debug: unmapping init [mem 0xffff8b8e3cc54000-0xffff8b8e3ffbffff] [ 12.324626] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 12.331846] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 12.350808] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 13.506325] Initialise system trusted keyrings [ 13.508177] Key type blacklist registered [ 13.514244] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 13.524903] zbud: loaded [ 13.667741] *** VALIDATE nfs *** [ 13.668569] *** VALIDATE nfs4 *** [ 13.669787] pstore: using deflate compression [ 13.672843] Platform Keyring initialized [ 13.789890] NET: Registered protocol family 38 [ 13.791450] Key type asymmetric registered [ 13.792907] Asymmetric key parser 'x509' registered [ 13.794510] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 13.798058] io scheduler mq-deadline registered [ 13.799516] io scheduler kyber registered [ 13.801283] io scheduler bfq registered [ 13.804552] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 13.807548] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 13.811486] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 13.814478] ACPI: Power Button [PWRF] [ 13.822356] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 13.829963] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 13.858838] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 13.887232] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 14.025310] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 14.057667] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 14.087844] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 14.094457] Non-volatile memory driver v1.3 [ 14.096280] Linux agpgart interface v0.103 [ 14.131480] virtio_blk virtio1: [vda] 134192 512-byte logical blocks (68.7 MB/65.5 MiB) [ 14.134833] vda: detected capacity change from 0 to 68706304 [ 14.150337] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 14.156892] vdb: detected capacity change from 0 to 1073741824 [ 14.216267] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 14.221550] vdc: detected capacity change from 0 to 2621440000 [ 14.254412] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 14.257532] vdd: detected capacity change from 0 to 2621440000 [ 14.284450] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 14.288185] vde: detected capacity change from 0 to 4294967296 [ 14.304762] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 14.308280] vdf: detected capacity change from 0 to 4294967296 [ 14.323474] libphy: Fixed MDIO Bus: probed [ 14.329922] usbcore: registered new interface driver usbserial_generic [ 14.362304] usbserial: USB Serial support registered for generic [ 14.364533] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 14.369778] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 14.371445] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 14.374163] mousedev: PS/2 mouse device common for all mice [ 14.378443] rtc_cmos 00:05: RTC can wake from S4 [ 14.382049] rtc_cmos 00:05: registered as rtc0 [ 14.383591] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 14.386826] intel_pstate: CPU model not supported [ 14.389545] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 14.393764] hid: raw HID events driver (C) Jiri Kosina [ 14.395502] usbcore: registered new interface driver usbhid [ 14.398511] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 14.400236] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 14.416296] usbhid: USB HID core driver [ 14.450256] drop_monitor: Initializing network drop monitor service [ 14.456885] Initializing XFRM netlink socket [ 14.462455] NET: Registered protocol family 10 [ 14.470680] Segment Routing with IPv6 [ 14.474194] NET: Registered protocol family 17 [ 14.478369] mpls_gso: MPLS GSO support [ 14.489302] RAS: Correctable Errors collector initialized. [ 14.493193] AVX version of gcm_enc/dec engaged. [ 14.496978] AES CTR mode by8 optimization enabled [ 14.706365] sched_clock: Marking stable (14706332848, 0)->(20225880491, -5519547643) [ 14.725946] registered taskstats version 1 [ 14.740436] Loading compiled-in X.509 certificates [ 14.746469] zswap: loaded using pool lzo/zbud [ 14.860184] Key type big_key registered [ 14.901330] Key type encrypted registered [ 14.903049] ima: No TPM chip found, activating TPM-bypass! [ 14.908174] ima: Allocated hash algorithm: sha1 [ 14.915370] ima: No architecture policies found [ 14.920168] evm: Initialising EVM extended attributes: [ 14.928281] evm: security.selinux [ 14.930412] evm: security.ima [ 14.935227] evm: security.capability [ 14.941703] evm: HMAC attrs: 0x1 [ 14.966236] rtc_cmos 00:05: setting system clock to 2026-01-22 04:02:11 UTC (1769054531) [ 15.000350] debug: unmapping init [mem 0xffffffffbbe03000-0xffffffffbbffffff] [ 15.037382] debug: unmapping init [mem 0xffffffffbab82000-0xffffffffbae58fff] [ 15.045195] Write protecting the kernel read-only data: 28672k [ 15.057413] debug: unmapping init [mem 0xffffffffb9203000-0xffffffffb93fffff] [ 15.068147] debug: unmapping init [mem 0xffffffffb9b14000-0xffffffffb9bfffff] [ 15.210874] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.235728] systemd[1]: Detected virtualization kvm. [ 15.240034] systemd[1]: Detected architecture x86-64. [ 15.241682] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 15.324904] systemd[1]: No hostname configured. [ 15.326270] systemd[1]: Set hostname to . [ 15.331354] random: systemd: uninitialized urandom read (16 bytes read) [ 15.345804] systemd[1]: Initializing machine ID from random generator. [ 15.625015] random: ln: uninitialized urandom read (6 bytes read) [ 16.093342] random: systemd: uninitialized urandom read (16 bytes read) [ 16.110460] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 16.149242] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 16.169761] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ 16.335142] urandom_read: 6 callbacks suppressed [ 16.335154] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 19.592367] device-mapper: uevent: version 1.0.3 [ 19.608700] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 23.401323] virtio_net virtio0 ens2: renamed from eth0 [ 23.759633] random: fast init done [ 24.469224] scsi host0: ata_piix [ 24.729200] scsi host1: ata_piix [ 24.733961] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 24.741199] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 29.498264] random: crng init done [ 33.343455] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 36.820440] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 41.955431] printk: systemd: 26 output lines suppressed due to ratelimiting [ 43.380478] SELinux: Disabled at runtime. [ 43.582526] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 43.607828] systemd[1]: Detected virtualization kvm. [ 43.614283] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 47.081859] systemd[1]: initrd-switch-root.service: Succeeded. [ 47.111828] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 47.147865] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 47.172906] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 47.189392] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 47.231614] systemd[1]: Starting Journal Service... Starting Journal Service... [ 47.270788] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ 47.845044] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 49.352780] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 51.165238] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 51.185334] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 52.357040] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 52.568650] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit)[ 62.229464] Key type dns_resolver registered [** ] A start job is running for Configur…only root support (15s / no limit) [* ] A start job is running for Configur…only root support (16s / no limit)[ 63.425992] NFS: Registering the id_resolver key type [ 63.437288] Key type id_resolver registered [ 63.440262] Key type id_legacy registered [** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg449-server login: [ 119.290369] libcfs: loading out-of-tree module taints kernel. [ 119.407690] Key type ._llcrypt registered [ 119.413104] Key type .llcrypt registered [ 119.587894] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_hostid [ 139.171664] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 141.018993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 141.035657] alg: No test for adler32 (adler32-zlib) [ 142.372892] Lustre: Lustre: Build Version: 2.17.50_63_ge38b609 [ 143.335219] LNet: Added LNI 192.168.204.149@tcp [8/256/0/180] [ 145.120301] Key type lgssc registered [ 147.023238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 166.199562] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 193.709422] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing load_modules_local [ 201.970338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 202.018856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.304192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 203.342163] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 203.407218] Lustre: lustre-MDT0000: new disk, initializing [ 203.481548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.494196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.231941] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.404173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 215.460906] Lustre: 6499:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.495502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 215.498983] Lustre: Skipped 1 previous similar message [ 215.556483] Lustre: lustre-MDT0001: new disk, initializing [ 215.591697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 215.604131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 215.609464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.397442] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.760980] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 227.419291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 227.572630] Lustre: lustre-OST0000: new disk, initializing [ 227.578244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 227.610237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.662766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.667894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.735724] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 231.079517] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.459542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 240.517526] Lustre: lustre-OST0001: new disk, initializing [ 240.522402] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.552373] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 244.654563] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.388318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 249.397730] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 249.418108] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 253.594710] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.653870] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 267.860766] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing check_logdir /tmp/testlogs/ [ 270.749383] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing yml_node [ 274.013154] Lustre: DEBUG MARKER: Client: 2.17.50.63 [ 275.766683] Lustre: DEBUG MARKER: MDS: 2.17.50.63 [ 277.721857] Lustre: DEBUG MARKER: OSS: 2.17.50.63 [ 279.034520] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Wed Jan 21 23:06:34 EST 2026 [ 292.311281] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 293.649269] Lustre: DEBUG MARKER: === replay-single: start setup 23:06:49 (1769054809) === [ 296.332203] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing check_config_client /mnt/lustre [ 309.289155] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 311.963906] Lustre: 13166:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 314.008406] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 317.490788] Lustre: DEBUG MARKER: === replay-single: finish setup 23:07:13 (1769054833) === [ 318.813723] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 23:07:14 (1769054834) [ 325.965860] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 328.663273] Lustre: Failing over lustre-MDT0000 [ 328.893675] LustreError: 14036:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 328.997747] Lustre: server umount lustre-MDT0000 complete [ 331.234602] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.240250] Lustre: Skipped 1 previous similar message [ 336.354903] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.410763] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 341.477329] LustreError: 12132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.518250] LustreError: 12132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 346.982853] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769054847/real 1769054847] req@ffff8b8d88aa4380 x1854988258212096/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769054863 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 346.983462] LustreError: 7781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.999663] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.042646] LustreError: 7781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 351.014184] LustreError: 7781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.049313] LustreError: 7781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 357.153770] LustreError: 10015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.197072] LustreError: 10015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 360.752633] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 360.760251] LDISKFS-fs (dm-0): recovery complete [ 360.828872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.547982] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.575215] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 371.681764] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8ebe5d6300 x1854988258224896/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 372.159847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 372.667198] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 377.465894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 377.576784] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 379.857828] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 391.143486] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 393.917775] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 404.638720] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 23:08:39 (1769054919) [ 407.456260] Lustre: Failing over lustre-OST0000 [ 407.529692] LustreError: 15482:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 407.536707] LustreError: 15482:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 407.730809] Lustre: server umount lustre-OST0000 complete [ 408.051243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.071767] Lustre: Skipped 3 previous similar messages [ 408.075716] LustreError: 8386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 408.109652] LustreError: 8386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 428.616637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 428.982429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.454360] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 430.988179] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 430.988879] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 430.988893] Lustre: Skipped 3 previous similar messages [ 436.984756] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 446.594197] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 448.362634] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 458.724226] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 23:09:34 (1769054974) [ 468.276571] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 470.650772] Lustre: Failing over lustre-MDT0000 [ 470.790272] LustreError: 17185:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 470.794791] LustreError: 17185:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 471.035063] Lustre: server umount lustre-MDT0000 complete [ 475.107933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.117869] LustreError: 12132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.123597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 475.127583] Lustre: Skipped 3 previous similar messages [ 475.153353] LustreError: 12132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 490.469255] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769054991/real 1769054991] req@ffff8b8d89045500 x1854988258289536/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769055007 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 490.506260] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 494.024540] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 494.028481] LDISKFS-fs (dm-0): recovery complete [ 494.046430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 500.706771] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8ebe5ead80 x1854988258297728/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 501.102507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.170977] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 506.343623] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 506.355413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 506.385176] Lustre: Skipped 1 previous similar message [ 508.860908] Lustre: lustre-MDT0000: Denying connection for new client adc973a5-ee43-40b0-8306-ad0ca550475c (at 192.168.204.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 513.978910] Lustre: lustre-MDT0000: Denying connection for new client adc973a5-ee43-40b0-8306-ad0ca550475c (at 192.168.204.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 514.001313] Lustre: Skipped 1 previous similar message [ 519.099759] Lustre: lustre-MDT0000: Denying connection for new client adc973a5-ee43-40b0-8306-ad0ca550475c (at 192.168.204.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 524.221357] Lustre: lustre-MDT0000: Denying connection for new client adc973a5-ee43-40b0-8306-ad0ca550475c (at 192.168.204.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 529.342932] Lustre: lustre-MDT0000: Denying connection for new client adc973a5-ee43-40b0-8306-ad0ca550475c (at 192.168.204.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:47 [ 539.578453] Lustre: lustre-MDT0000: Denying connection for new client adc973a5-ee43-40b0-8306-ad0ca550475c (at 192.168.204.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:36 [ 539.594141] Lustre: Skipped 1 previous similar message [ 560.068421] Lustre: lustre-MDT0000: Denying connection for new client adc973a5-ee43-40b0-8306-ad0ca550475c (at 192.168.204.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:16 [ 560.093250] Lustre: Skipped 3 previous similar messages [ 576.500166] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 576.506219] Lustre: 17847:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 1e073c4d-89b6-4f4e-bb5e-0e4164dcb0bf@ [ 576.518507] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 576.547820] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 576.548833] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 576.564815] Lustre: Skipped 2 previous similar messages [ 576.604500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 576.605018] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 590.426917] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 23:11:45 (1769055105) [ 598.052647] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 600.016773] Lustre: Failing over lustre-MDT0000 [ 600.236564] LustreError: 18910:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 600.243819] LustreError: 18910:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 600.718130] Lustre: server umount lustre-MDT0000 complete [ 602.080617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 602.090513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.126490] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.152653] LustreError: 6510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 620.026841] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769055120/real 1769055120] req@ffff8b8ebac70380 x1854988258350720/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769055136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 620.093419] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 622.504573] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 622.506746] LDISKFS-fs (dm-0): recovery complete [ 622.517025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.641953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.719890] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 635.876799] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 635.883269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 636.623273] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 639.281890] Lustre: lustre-MDT0000: Denying connection for new client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 639.305583] Lustre: Skipped 3 previous similar messages [ 705.500208] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 705.503047] Lustre: 19566:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client adc973a5-ee43-40b0-8306-ad0ca550475c@ [ 705.553354] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 705.579564] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 705.583602] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 705.605952] Lustre: Skipped 2 previous similar messages [ 705.697496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 705.698439] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 717.436383] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 23:13:52 (1769055232) [ 727.756937] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 730.392745] Lustre: Failing over lustre-MDT0000 [ 730.527856] LustreError: 20635:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 730.538131] LustreError: 20635:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 730.708388] Lustre: server umount lustre-MDT0000 complete [ 731.618411] LustreError: 12132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 731.640265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 731.657097] LustreError: 12132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 731.709272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 731.746542] Lustre: Skipped 3 previous similar messages [ 749.537042] Lustre: 3653:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769055249/real 1769055249] req@ffff8b8d8905c700 x1854988258408320/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769055265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 749.558578] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 755.222785] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 755.226251] LDISKFS-fs (dm-0): recovery complete [ 755.234460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.125921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.165834] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 761.878324] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 765.425445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 765.598082] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 765.662454] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 765.665383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 766.168344] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 775.565684] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 778.320862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 789.573128] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 23:15:04 (1769055304) [ 798.266574] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 800.865213] Lustre: Failing over lustre-MDT0000 [ 801.066270] LustreError: 22469:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 801.070103] LustreError: 22469:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 801.280065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 801.294491] Lustre: Skipped 3 previous similar messages [ 801.422277] Lustre: server umount lustre-MDT0000 complete [ 817.107256] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769055317/real 1769055317] req@ffff8b8d877a3100 x1854988258446848/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769055333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 817.141064] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 826.403370] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 826.414308] LDISKFS-fs (dm-0): recovery complete [ 826.433866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.857414] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 828.871695] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 833.050652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 833.056772] Lustre: Skipped 3 previous similar messages [ 833.193172] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 833.251451] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 833.263037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 833.415611] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 842.754191] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 844.852845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 855.458952] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 23:16:10 (1769055370) [ 863.829713] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 866.173134] Lustre: Failing over lustre-MDT0000 [ 866.312244] LustreError: 24301:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 866.323184] LustreError: 24301:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 866.533767] Lustre: server umount lustre-MDT0000 complete [ 868.841976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.866316] Lustre: Skipped 4 previous similar messages [ 885.216185] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769055385/real 1769055385] req@ffff8b8ea7e76300 x1854988258484736/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769055401 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 885.247715] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.820664] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 890.830820] LDISKFS-fs (dm-0): recovery complete [ 890.839817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.457876] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8d89cbed80 x1854988258493440/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 895.885403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 895.896669] Lustre: Skipped 1 previous similar message [ 895.949157] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 896.949151] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 901.105822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 901.124303] Lustre: Skipped 3 previous similar messages [ 901.276138] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 901.356363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 901.357465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 901.732474] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 910.528114] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 912.561339] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 924.007566] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 23:17:19 (1769055439) [ 932.206906] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 934.443222] Lustre: Failing over lustre-MDT0000 [ 934.694289] LustreError: 26138:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 934.702537] LustreError: 26138:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 934.919853] Lustre: server umount lustre-MDT0000 complete [ 936.937320] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 936.962570] Lustre: Skipped 3 previous similar messages [ 953.312198] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769055453/real 1769055453] req@ffff8b8d884a2300 x1854988258522880/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769055469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 953.328791] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 958.066567] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 958.070852] LDISKFS-fs (dm-0): recovery complete [ 958.087037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.528981] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8e9fe66a00 x1854988258531200/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 962.925401] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 965.051226] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 967.894489] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 968.453089] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 968.531651] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 968.534379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 976.181842] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 978.567492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 986.954274] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 23:18:22 (1769055502) [ 996.614630] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 999.729311] Lustre: Failing over lustre-MDT0000 [ 999.970494] LustreError: 27973:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 999.978415] LustreError: 27973:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1000.270131] Lustre: server umount lustre-MDT0000 complete [ 1000.926872] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.958377] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 138 previous similar messages [ 1004.003307] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.024669] Lustre: Skipped 3 previous similar messages [ 1019.858163] Lustre: 3653:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769055520/real 1769055520] req@ffff8b8d88498380 x1854988258561536/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769055536 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1019.907875] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.828986] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1023.836958] LDISKFS-fs (dm-0): recovery complete [ 1023.857102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1030.116602] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795ce20f0 [ 1030.134528] Lustre: MGC192.168.204.149@tcp: Connection restored to 0@lo (at 0@lo) [ 1030.139908] Lustre: Skipped 7 previous similar messages [ 1030.623634] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1031.405622] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1035.899279] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1035.933340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1035.935405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1036.882356] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1046.516001] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1048.036864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1056.572964] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 23:19:32 (1769055572) [ 1057.608875] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1057.610703] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1057.621967] LustreError: 6508:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8d8849ad80 x1854988238347008/t38654705666(0) o35->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:740/0 lens 392/456 e 0 to 0 dl 1769055585 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1066.424915] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1068.493434] Lustre: Failing over lustre-MDT0000 [ 1068.821473] Lustre: server umount lustre-MDT0000 complete [ 1087.972525] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769055588/real 1769055588] req@ffff8b8e9ff5bb80 x1854988258599936/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769055604 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1088.000146] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.908596] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1089.913571] LDISKFS-fs (dm-0): recovery complete [ 1089.951602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.210331] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795ce25c7 [ 1098.565876] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1103.945293] Lustre: 6508:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ebac79180 x1854988238347008/t38654705666(0) o35->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:31/0 lens 392/456 e 0 to 0 dl 1769055631 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1104.015086] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1104.026192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1104.062318] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1112.228687] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1113.904480] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1121.490107] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 23:20:37 (1769055637) [ 1129.851891] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1132.903231] Lustre: Failing over lustre-MDT0000 [ 1133.144638] LustreError: 31693:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1133.152307] LustreError: 31693:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 1133.289917] Lustre: server umount lustre-MDT0000 complete [ 1134.562300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1134.581672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.608495] Lustre: Skipped 8 previous similar messages [ 1158.186864] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1158.193097] LDISKFS-fs (dm-0): recovery complete [ 1158.242908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.547151] LustreError: 32312:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1160.554325] LustreError: 32312:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8b8e9ff5ed80 x1854988258640256/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1769055677 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1160.566426] LustreError: 32312:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1161.207681] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8ebb8b8e00 x1854988258643712/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1161.684974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.694240] Lustre: Skipped 3 previous similar messages [ 1161.743735] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1163.437221] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1163.440629] Lustre: Skipped 1 previous similar message [ 1166.930970] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1166.945534] Lustre: Skipped 1 previous similar message [ 1166.994453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1166.996352] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1167.156700] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1175.748159] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1177.417892] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1185.419682] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 23:21:41 (1769055701) [ 1192.075645] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1192.940459] Lustre: *** cfs_fail_loc=114, val=0*** [ 1195.717715] Lustre: Failing over lustre-MDT0000 [ 1196.067343] Lustre: server umount lustre-MDT0000 complete [ 1197.572346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1218.486923] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1218.489866] LDISKFS-fs (dm-0): recovery complete [ 1218.499383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.169969] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8d88499880 x1854988258676864/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1229.175123] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1229.924830] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1229.930065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1237.653083] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1239.286473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1247.077659] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 23:22:42 (1769055762) [ 1254.220543] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1255.084599] Lustre: *** cfs_fail_loc=128, val=0*** [ 1257.463347] Lustre: Failing over lustre-MDT0000 [ 1257.845977] Lustre: server umount lustre-MDT0000 complete [ 1260.518772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1276.898075] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769055777/real 1769055777] req@ffff8b8ea7c4c000 x1854988258703232/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769055793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1276.939530] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1276.946988] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1276.951965] LustreError: Skipped 2 previous similar messages [ 1280.587945] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1280.591301] LDISKFS-fs (dm-0): recovery complete [ 1280.602129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.160893] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8e83f1aa00 x1854988258712320/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1288.582339] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1288.591351] Lustre: Skipped 1 previous similar message [ 1292.740818] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1293.807867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1293.826024] Lustre: Skipped 17 previous similar messages [ 1293.903392] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1293.908451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1300.915128] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1302.504365] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1311.089530] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 23:23:46 (1769055826) [ 1318.446272] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1320.398510] Lustre: Failing over lustre-MDT0000 [ 1320.627130] Lustre: server umount lustre-MDT0000 complete [ 1345.495600] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1345.501159] LDISKFS-fs (dm-0): recovery complete [ 1345.517393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.137685] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795ce3b5a [ 1356.187322] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1357.067507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1357.085181] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1364.934215] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1366.601488] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1375.457947] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 23:24:50 (1769055890) [ 1382.633410] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1384.480536] Lustre: Failing over lustre-MDT0000 [ 1384.794715] Lustre: server umount lustre-MDT0000 complete [ 1409.452658] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1409.457212] LDISKFS-fs (dm-0): recovery complete [ 1409.470716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.113072] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8d88883b80 x1854988258784256/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1418.934939] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1420.100308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1420.102993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1426.469840] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1428.633109] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1436.581647] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 23:25:52 (1769055952) [ 1445.180940] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1454.703656] Lustre: Failing over lustre-MDT0000 [ 1454.864768] LustreError: 41055:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1454.868223] LustreError: 41055:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 1455.082534] Lustre: server umount lustre-MDT0000 complete [ 1455.586042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.618389] Lustre: Skipped 17 previous similar messages [ 1477.989944] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1477.995078] LDISKFS-fs (dm-0): recovery complete [ 1478.016487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1482.523925] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1482.535416] Lustre: Skipped 2 previous similar messages [ 1484.175787] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1484.184955] Lustre: Skipped 4 previous similar messages [ 1486.973349] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1491.603066] Lustre: lustre-MDT0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 1491.616200] Lustre: Skipped 4 previous similar messages [ 1491.662106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1491.662588] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1496.094683] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1498.074311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1522.838676] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 23:27:18 (1769056038) [ 1531.561804] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1533.898413] Lustre: Failing over lustre-MDT0000 [ 1533.930090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1534.423524] Lustre: server umount lustre-MDT0000 complete [ 1535.427152] LustreError: 12132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.451553] LustreError: 12132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 277 previous similar messages [ 1555.424253] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769056055/real 1769056055] req@ffff8b8e9ff69180 x1854988258916608/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769056071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1555.441031] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1555.445548] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1555.451520] LustreError: Skipped 3 previous similar messages [ 1558.028556] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1558.030763] LDISKFS-fs (dm-0): recovery complete [ 1558.038110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.309854] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1570.454768] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1570.460371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1577.203518] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1579.058161] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1591.103279] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 23:28:26 (1769056106) [ 1599.835678] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1601.992794] Lustre: Failing over lustre-MDT0000 [ 1602.318196] Lustre: server umount lustre-MDT0000 complete [ 1627.994444] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1628.000793] LDISKFS-fs (dm-0): recovery complete [ 1628.011303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1636.340446] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1637.580973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1637.584026] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1645.937819] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1648.328733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1658.116596] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 23:29:33 (1769056173) [ 1667.478693] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1669.103503] Lustre: Failing over lustre-MDT0000 [ 1669.360193] Lustre: server umount lustre-MDT0000 complete [ 1692.598561] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1692.603611] LDISKFS-fs (dm-0): recovery complete [ 1692.617942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1699.142304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1699.145535] Lustre: Skipped 7 previous similar messages [ 1704.047291] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1704.562157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1704.565674] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1711.700907] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1713.204211] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1721.930848] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 23:30:37 (1769056237) [ 1729.516952] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1731.923413] Lustre: Failing over lustre-MDT0000 [ 1732.419036] Lustre: server umount lustre-MDT0000 complete [ 1735.160292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1755.584927] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1755.588434] LDISKFS-fs (dm-0): recovery complete [ 1755.596952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1761.249331] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8ebae8ad80 x1854988259040128/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1761.727500] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1761.747663] Lustre: Skipped 3 previous similar messages [ 1766.434688] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1767.128624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1767.129730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1775.696802] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1777.762747] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1787.687086] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 23:31:43 (1769056303) [ 1795.198680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1797.522245] Lustre: Failing over lustre-MDT0000 [ 1797.604351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.617284] Lustre: Skipped 3 previous similar messages [ 1797.893856] Lustre: server umount lustre-MDT0000 complete [ 1822.751744] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1822.756336] LDISKFS-fs (dm-0): recovery complete [ 1822.770215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.988318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1835.007825] Lustre: Skipped 32 previous similar messages [ 1835.176350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1835.176445] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1835.571957] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1843.212767] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1845.672696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1855.372588] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 23:32:50 (1769056370) [ 1864.455346] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1866.760618] Lustre: Failing over lustre-MDT0000 [ 1867.012877] Lustre: server umount lustre-MDT0000 complete [ 1892.064094] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1892.066206] LDISKFS-fs (dm-0): recovery complete [ 1892.098848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.907361] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1903.284668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1903.285172] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1912.473430] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1914.348866] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1924.394661] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 23:33:59 (1769056439) [ 1933.260753] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1935.418179] Lustre: Failing over lustre-MDT0000 [ 1935.858903] Lustre: server umount lustre-MDT0000 complete [ 1959.327653] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1959.331230] LDISKFS-fs (dm-0): recovery complete [ 1959.336798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.517637] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795cf03ba [ 1969.239256] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1970.401458] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1970.407624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1977.688554] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1979.237203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1987.575501] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 23:35:02 (1769056502) [ 1994.870667] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1997.095240] Lustre: Failing over lustre-MDT0000 [ 1997.259697] LustreError: 55764:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1997.267499] LustreError: 55764:0:(obd_class.h:479:obd_check_dev()) Skipped 63 previous similar messages [ 1997.784394] Lustre: server umount lustre-MDT0000 complete [ 2000.867779] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.896946] Lustre: Skipped 32 previous similar messages [ 2023.996959] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2024.000034] LDISKFS-fs (dm-0): recovery complete [ 2024.028499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.489943] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8ebb664e00 x1854988259186688/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2027.973716] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2027.985669] Lustre: Skipped 7 previous similar messages [ 2033.226988] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2033.249029] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 2033.258652] Lustre: Skipped 7 previous similar messages [ 2033.356032] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 2033.366972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 2042.274309] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2044.497840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2056.614594] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 23:36:11 (1769056571) [ 2065.661428] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2067.974794] Lustre: Failing over lustre-MDT0000 [ 2068.208377] Lustre: server umount lustre-MDT0000 complete [ 2085.344424] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769056585/real 1769056585] req@ffff8b8ebbc12680 x1854988259215872/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769056601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2085.381635] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2085.394677] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.408124] LustreError: Skipped 7 previous similar messages [ 2091.762617] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2091.769426] LDISKFS-fs (dm-0): recovery complete [ 2091.791960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.137656] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2100.349957] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 2100.350300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 2106.808268] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2108.962991] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2118.240702] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 23:37:13 (1769056633) [ 2126.735331] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2129.001519] Lustre: Failing over lustre-MDT0000 [ 2130.915500] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2131.202739] Lustre: server umount lustre-MDT0000 complete [ 2136.034202] LustreError: 12132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.049205] LustreError: 12132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 307 previous similar messages [ 2154.457808] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2154.464960] LDISKFS-fs (dm-0): recovery complete [ 2154.473666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.321547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:897) [ 2162.323816] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 2162.470631] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2171.018502] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2173.065925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2182.236608] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 23:38:17 (1769056697) [ 2192.601881] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2195.455405] Lustre: Failing over lustre-MDT0000 [ 2196.020662] Lustre: server umount lustre-MDT0000 complete [ 2221.945805] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2221.950557] LDISKFS-fs (dm-0): recovery complete [ 2221.965043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.941388] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2229.342570] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2229.344026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2236.001682] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2238.145165] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2248.753945] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 23:39:23 (1769056763) [ 2258.323359] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2261.185702] Lustre: Failing over lustre-MDT0000 [ 2261.458720] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 2261.467027] Lustre: Skipped 3 previous similar messages [ 2261.706320] Lustre: server umount lustre-MDT0000 complete [ 2265.061567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2286.758578] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2286.762564] LDISKFS-fs (dm-0): recovery complete [ 2286.770841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.661621] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795cf1f04 [ 2291.057506] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2291.071586] Lustre: Skipped 7 previous similar messages [ 2296.498908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 2296.502797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 2296.646507] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2305.874483] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2308.150210] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2317.567891] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 23:40:32 (1769056832) [ 2325.983571] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2328.098419] Lustre: Failing over lustre-MDT0000 [ 2328.558158] Lustre: server umount lustre-MDT0000 complete [ 2352.729105] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2352.731790] LDISKFS-fs (dm-0): recovery complete [ 2352.745086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.242541] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8ea7a47100 x1854988259379072/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2358.287426] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 47 previous similar messages [ 2358.765509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.768466] Lustre: Skipped 9 previous similar messages [ 2364.068199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:993) [ 2364.069158] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 2365.607431] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2374.445330] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2376.463985] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2387.922309] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 23:41:42 (1769056902) [ 2397.040680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2399.253860] Lustre: Failing over lustre-MDT0000 [ 2399.624193] Lustre: server umount lustre-MDT0000 complete [ 2400.224548] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2424.065800] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2424.069071] LDISKFS-fs (dm-0): recovery complete [ 2424.086646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2430.443766] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795cf2a3a [ 2435.892487] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2436.107486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2436.130065] Lustre: Skipped 38 previous similar messages [ 2436.356658] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2436.356475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2444.627336] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2446.469862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2455.812572] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 23:42:51 (1769056971) [ 2463.387617] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2465.549934] Lustre: Failing over lustre-MDT0000 [ 2465.852757] Lustre: server umount lustre-MDT0000 complete [ 2489.717697] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2489.719731] LDISKFS-fs (dm-0): recovery complete [ 2489.756753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.138302] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2499.228168] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2499.228509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2508.569331] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2510.301203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2519.929119] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 23:43:54 (1769057034) [ 2528.413543] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2530.196890] Lustre: Failing over lustre-MDT0000 [ 2530.275259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2530.410778] Lustre: server umount lustre-MDT0000 complete [ 2554.107063] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2554.111654] LDISKFS-fs (dm-0): recovery complete [ 2554.126472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.792346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 2567.794392] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 2568.915322] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2578.449306] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2580.118338] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2588.416795] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 23:45:03 (1769057103) [ 2592.723923] Lustre: 71917:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 61c43a80-9ad2-4b8b-bb68-67fc6884d641 at adminstrative request [ 2597.254166] Lustre: Failing over lustre-MDT0000 [ 2597.406238] LustreError: 72062:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2597.414311] LustreError: 72062:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 2597.553697] Lustre: server umount lustre-MDT0000 complete [ 2617.417633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.003491] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795cf3bba [ 2629.696851] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2630.739880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2630.740793] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 2637.624607] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2639.594633] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2645.391571] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2652.712535] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2658.312669] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 23:46:14 (1769057174) [ 2659.231426] Lustre: 73770:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 61c43a80-9ad2-4b8b-bb68-67fc6884d641 at adminstrative request [ 2668.025256] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 23:46:23 (1769057183) [ 2674.544389] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2676.560277] Lustre: Failing over lustre-MDT0000 [ 2676.712839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2676.719429] LustreError: Skipped 2 previous similar messages [ 2676.723036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.731825] Lustre: Skipped 37 previous similar messages [ 2676.740411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2676.745969] Lustre: Skipped 2 previous similar messages [ 2676.885260] Lustre: server umount lustre-MDT0000 complete [ 2698.209106] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769057198/real 1769057198] req@ffff8b8ebaf1aa00 x1854988259563264/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769057214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2698.232321] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2698.242905] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2698.252446] LustreError: Skipped 8 previous similar messages [ 2698.678161] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2698.679818] LDISKFS-fs (dm-0): recovery complete [ 2698.689984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.454167] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795cf415c [ 2708.925641] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2708.941461] Lustre: Skipped 9 previous similar messages [ 2712.792973] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2714.165072] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 2714.172444] Lustre: Skipped 9 previous similar messages [ 2714.206131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 2714.206194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 2719.354546] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2720.696861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2728.376487] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 23:47:24 (1769057244) [ 2735.501951] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2737.367097] Lustre: Failing over lustre-MDT0000 [ 2737.772627] Lustre: server umount lustre-MDT0000 complete [ 2739.649954] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.670430] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 298 previous similar messages [ 2759.092757] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2759.095439] LDISKFS-fs (dm-0): recovery complete [ 2759.101824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.767563] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2772.056189] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2772.062936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2777.841831] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2779.696907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2787.744033] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 23:48:23 (1769057303) [ 2794.763187] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2796.797134] Lustre: Failing over lustre-MDT0000 [ 2797.114965] Lustre: server umount lustre-MDT0000 complete [ 2819.875527] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2819.880927] LDISKFS-fs (dm-0): recovery complete [ 2819.893377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.670643] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2829.902817] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2829.906400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2835.892861] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2837.555908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2846.821122] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 23:49:22 (1769057362) [ 2854.956909] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2856.829741] Lustre: Failing over lustre-MDT0000 [ 2857.036380] Lustre: server umount lustre-MDT0000 complete [ 2877.469721] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2877.472461] LDISKFS-fs (dm-0): recovery complete [ 2877.479217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.637828] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795cf51e7 [ 2890.664337] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2892.309501] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2892.318047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2897.197867] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2898.530981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2905.772737] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 23:50:21 (1769057421) [ 2911.654384] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2913.200266] Lustre: Failing over lustre-MDT0000 [ 2915.395560] Lustre: server umount lustre-MDT0000 complete [ 2936.304798] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2936.306743] LDISKFS-fs (dm-0): recovery complete [ 2936.313209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.482090] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8d8aa29180 x1854988259705216/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2944.501246] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [ 2944.730891] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2944.737896] Lustre: Skipped 9 previous similar messages [ 2947.613599] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2950.189112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 2950.189499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 2953.517204] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2954.600804] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2961.044766] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 23:51:16 (1769057476) [ 2967.499658] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2969.278344] Lustre: Failing over lustre-MDT0000 [ 2969.476674] Lustre: server umount lustre-MDT0000 complete [ 2989.749722] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2989.751744] LDISKFS-fs (dm-0): recovery complete [ 2989.767033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2997.484631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.487605] Lustre: Skipped 9 previous similar messages [ 3001.382784] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3003.055726] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 3003.059486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 3013.618872] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3015.798805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3031.596136] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 23:52:25 (1769057545) [ 3044.134700] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3046.504165] Lustre: Failing over lustre-MDT0000 [ 3046.784923] Lustre: server umount lustre-MDT0000 complete [ 3071.389946] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3071.396384] LDISKFS-fs (dm-0): recovery complete [ 3071.406604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.045511] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3080.167158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3080.174466] Lustre: Skipped 42 previous similar messages [ 3080.352830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 3080.357963] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 3087.600628] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3089.306486] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3098.917533] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 23:53:34 (1769057614) [ 3106.461303] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3108.617933] Lustre: Failing over lustre-MDT0000 [ 3109.208319] Lustre: server umount lustre-MDT0000 complete [ 3134.320955] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3134.322960] LDISKFS-fs (dm-0): recovery complete [ 3134.343325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3136.483783] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795cf68d8 [ 3142.275837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 3142.287914] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 3143.188785] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3153.096074] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3154.957574] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3165.331668] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 23:54:40 (1769057680) [ 3174.507852] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3177.363620] Lustre: Failing over lustre-MDT0000 [ 3177.645022] Lustre: server umount lustre-MDT0000 complete [ 3204.595457] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3204.599754] LDISKFS-fs (dm-0): recovery complete [ 3204.616675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.724473] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3214.983496] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3214.983933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3224.590512] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3226.543800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3235.460447] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 23:55:51 (1769057751) [ 3243.454168] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3245.540423] Lustre: Failing over lustre-MDT0000 [ 3245.697537] LustreError: 90969:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3245.702801] LustreError: 90969:0:(obd_class.h:479:obd_check_dev()) Skipped 79 previous similar messages [ 3246.310726] Lustre: server umount lustre-MDT0000 complete [ 3250.656952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3250.671398] LustreError: Skipped 6 previous similar messages [ 3270.252872] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3270.256480] LDISKFS-fs (dm-0): recovery complete [ 3270.263995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.279585] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3282.490542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 3282.496628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3291.233620] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3293.276099] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3302.551876] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 23:56:58 (1769057818) [ 3312.266776] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3314.673190] Lustre: Failing over lustre-MDT0000 [ 3315.001955] Lustre: server umount lustre-MDT0000 complete [ 3318.244399] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.262141] Lustre: Skipped 41 previous similar messages [ 3334.624362] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769057834/real 1769057834] req@ffff8b8ebaec1c00 x1854988259918080/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769057850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3334.665055] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 3334.680474] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3334.695371] LustreError: Skipped 9 previous similar messages [ 3339.721035] LustreError: 10015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3339.754257] LustreError: 10015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 333 previous similar messages [ 3339.800240] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3339.802993] LDISKFS-fs (dm-0): recovery complete [ 3339.809421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3346.922440] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3346.942885] Lustre: Skipped 9 previous similar messages [ 3350.548046] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3350.556530] Lustre: Skipped 9 previous similar messages [ 3350.622856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 3350.623624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 3350.624795] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3360.145702] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3362.175605] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3372.697412] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 23:58:07 (1769057887) [ 3373.917498] Lustre: 94194:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 61c43a80-9ad2-4b8b-bb68-67fc6884d641 at adminstrative request [ 3385.852234] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 23:58:20 (1769057900) [ 3393.768815] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3396.202487] Lustre: Failing over lustre-MDT0000 [ 3396.508425] Lustre: server umount lustre-MDT0000 complete [ 3409.922510] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3409.927137] LDISKFS-fs (dm-0): recovery complete [ 3409.958192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3410.544885] Lustre: lustre-MDT0000: Aborting client recovery [ 3410.547057] LustreError: 95477:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3410.554502] Lustre: 95513:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3410.561473] Lustre: 95513:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 61c43a80-9ad2-4b8b-bb68-67fc6884d641@ [ 3410.575866] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3410.589757] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3410.609392] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3410.697618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1479 to 0x280000401:1505) [ 3410.703589] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1479 to 0x2c0000401:1505) [ 3415.528359] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3416.235700] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3440.045350] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 23:59:15 (1769057955) [ 3449.242968] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3451.168336] Lustre: Failing over lustre-MDT0000 [ 3451.412133] Lustre: server umount lustre-MDT0000 complete [ 3465.505444] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3465.508773] LDISKFS-fs (dm-0): recovery complete [ 3465.532900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3466.066613] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3466.090910] Lustre: lustre-MDT0000: Aborting client recovery [ 3466.094288] LustreError: 97335:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3466.099347] Lustre: 97371:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3466.104594] Lustre: 97371:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3466.109301] Lustre: 97371:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3466.115947] Lustre: 97371:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3466.120955] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3466.127157] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3466.141554] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3466.177865] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 3466.185529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 3471.339190] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3471.683986] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3486.653883] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3486.662932] Lustre: Skipped 1 previous similar message [ 3494.088739] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 00:00:09 (1769058009) [ 3502.039797] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3504.295281] Lustre: Failing over lustre-MDT0000 [ 3504.532606] Lustre: server umount lustre-MDT0000 complete [ 3519.918896] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3519.928834] LDISKFS-fs (dm-0): recovery complete [ 3519.945890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3520.298488] Lustre: lustre-MDT0000: Aborting client recovery [ 3520.306527] LustreError: 99195:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3520.310500] Lustre: 99231:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3520.317173] Lustre: 99231:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3520.323451] Lustre: 99231:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3520.331186] Lustre: 99231:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3520.336678] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3520.346925] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3520.359712] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3520.399878] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 3520.401085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 3525.638985] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3526.089088] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3550.173396] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 00:01:05 (1769058065) [ 3551.430837] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3551.440361] LustreError: 10015:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8ebabe6300 x1854988239449088/t201863462916(0) o36->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:213/0 lens 512/456 e 0 to 0 dl 1769058078 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3555.588544] Lustre: Failing over lustre-MDT0000 [ 3555.905846] Lustre: server umount lustre-MDT0000 complete [ 3565.649403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.186116] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3566.186409] Lustre: lustre-MDT0000: Aborting client recovery [ 3566.194649] Lustre: Skipped 15 previous similar messages [ 3566.204732] LustreError: 100524:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3566.214820] Lustre: 100560:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3566.223514] Lustre: 100560:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3566.229416] Lustre: 100560:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3566.236542] Lustre: 100560:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3566.243802] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3566.254771] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3566.283688] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3566.345372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 3566.349909] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1601) [ 3571.201120] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3571.722849] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3595.255735] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3596.911495] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 00:01:52 (1769058112) [ 3605.017968] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3607.881530] Lustre: Failing over lustre-MDT0000 [ 3607.998441] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 3608.126361] Lustre: server umount lustre-MDT0000 complete [ 3621.710760] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3621.716398] LDISKFS-fs (dm-0): recovery complete [ 3621.736604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3622.150394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3622.154538] Lustre: Skipped 9 previous similar messages [ 3622.191226] Lustre: lustre-MDT0000: Aborting client recovery [ 3622.196928] LustreError: 102479:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3622.202414] Lustre: 102516:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3622.209723] Lustre: 102516:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3622.218762] Lustre: 102516:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3622.228798] Lustre: 102516:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3622.234713] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3622.244992] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3622.259723] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3622.323440] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1633) [ 3622.342237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 3627.501298] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3628.596613] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3653.861136] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:02:49 (1769058169) [ 3687.852422] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3690.378718] Lustre: Failing over lustre-MDT0000 [ 3691.231737] Lustre: server umount lustre-MDT0000 complete [ 3717.201858] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3717.203817] LDISKFS-fs (dm-0): recovery complete [ 3717.230828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3719.648905] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8eba9ff800 x1854988260215296/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3725.308342] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3725.330085] Lustre: Skipped 40 previous similar messages [ 3725.575931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3725.579621] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3725.581625] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3735.707161] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3737.706986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3763.041537] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:04:37 (1769058277) [ 3789.526733] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3800.263512] Lustre: Failing over lustre-MDT0000 [ 3800.730798] Lustre: server umount lustre-MDT0000 complete [ 3824.881443] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3824.885756] LDISKFS-fs (dm-0): recovery complete [ 3824.897990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.682504] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795d1eda3 [ 3834.143082] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3837.323838] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3837.324280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3842.581161] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3844.453611] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3867.527843] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 00:06:22 (1769058382) [ 3869.743421] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3870.829904] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3870.841505] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3878.184380] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 00:06:33 (1769058393) [ 3905.361924] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3918.142562] Lustre: Failing over lustre-OST0000 [ 3918.211170] LustreError: 109029:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3918.219348] LustreError: 109029:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 3918.306935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.317982] Lustre: Skipped 30 previous similar messages [ 3918.349233] Lustre: server umount lustre-OST0000 complete [ 3941.824583] LustreError: 8386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3941.847182] LustreError: 8386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 152 previous similar messages [ 3943.782079] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3943.786940] LDISKFS-fs (dm-2): recovery complete [ 3943.797886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3951.393740] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4014.570585] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 00:08:50 (1769058530) [ 4022.492536] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4025.492362] Lustre: Failing over lustre-MDT0000 [ 4025.799376] Lustre: server umount lustre-MDT0000 complete [ 4047.328144] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769058547/real 1769058547] req@ffff8b8ebaf18700 x1854988260720768/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769058563 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4047.362106] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4047.373191] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4047.380637] LustreError: Skipped 7 previous similar messages [ 4049.130274] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4049.133483] LDISKFS-fs (dm-0): recovery complete [ 4049.142560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4058.556624] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4058.566296] Lustre: Skipped 3 previous similar messages [ 4062.309917] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4062.316385] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4062.355338] Lustre: Skipped 3 previous similar messages [ 4062.422921] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 4062.425148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 4071.302406] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4073.209746] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4077.544319] LustreError: 111689:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4077.548782] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4078.624677] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 4093.552643] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 00:10:08 (1769058608) [ 4098.391482] LustreError: 6507:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4103.653179] LustreError: 6507:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4103.671539] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnecting [ 4103.779747] LustreError: 8387:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 4105.826906] LustreError: 6507:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4111.328353] LustreError: 6507:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4111.338232] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnecting [ 4113.933870] LustreError: 10015:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4119.008245] LustreError: 10015:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4119.011387] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnecting [ 4121.380043] LustreError: 10015:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4126.688181] LustreError: 10015:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4128.727664] LustreError: 12132:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4133.857269] LustreError: 12132:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4133.872958] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnecting [ 4133.883698] Lustre: Skipped 1 previous similar message [ 4143.383025] LustreError: 12132:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4143.395799] LustreError: 12132:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4148.708567] LustreError: 12132:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4148.715437] LustreError: 12132:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4156.384773] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnecting [ 4156.392015] Lustre: Skipped 2 previous similar messages [ 4165.599487] LustreError: 6505:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4165.609043] LustreError: 6505:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4170.720615] LustreError: 6505:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4170.737108] LustreError: 6505:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4179.678967] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 00:11:34 (1769058694) [ 4181.718327] LustreError: 10015:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4191.163233] Lustre: lustre-MDT0000: Export ffff8b8ea7cfa000 already connecting from 192.168.204.49@tcp [ 4193.212623] Lustre: lustre-MDT0000: Export ffff8b8ea7cfa000 already connecting from 192.168.204.49@tcp [ 4196.217916] Lustre: lustre-MDT0000: Export ffff8b8ea7cfa000 already connecting from 192.168.204.49@tcp [ 4201.404919] Lustre: lustre-MDT0000: Export ffff8b8ea7cfa000 already connecting from 192.168.204.49@tcp [ 4206.526829] Lustre: lustre-MDT0000: Export ffff8b8ea7cfa000 already connecting from 192.168.204.49@tcp [ 4206.540773] Lustre: Skipped 1 previous similar message [ 4216.763112] Lustre: lustre-MDT0000: Export ffff8b8ea7cfa000 already connecting from 192.168.204.49@tcp [ 4216.777868] Lustre: Skipped 1 previous similar message [ 4221.768235] LustreError: 10015:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4221.773342] Lustre: 10015:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b8d88498e00 x1854988242112768/t0(0) o38->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1769058718 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4221.893476] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnecting [ 4221.906421] Lustre: Skipped 3 previous similar messages [ 4221.909173] LustreError: 6507:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4246.463708] Lustre: lustre-MDT0000: Export ffff8b8ea7cfa000 already connecting from 192.168.204.49@tcp [ 4261.984757] LustreError: 6507:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4261.991830] Lustre: 6507:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b8ea7acad80 x1854988242117376/t0(0) o38->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1769058758 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4266.938062] LustreError: 6505:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4291.518286] Lustre: lustre-MDT0000: Export ffff8b8ea7cfa000 already connecting from 192.168.204.49@tcp [ 4291.527549] Lustre: Skipped 4 previous similar messages [ 4306.953918] LustreError: 6505:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4306.972069] Lustre: 6505:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b8ebe55bb80 x1854988242120960/t0(0) o38->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1769058803 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4311.995712] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnecting [ 4312.009299] Lustre: Skipped 1 previous similar message [ 4312.020276] LustreError: 12132:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4352.072149] LustreError: 12132:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4352.080170] Lustre: 12132:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b8ea7ac9500 x1854988242124672/t0(0) o38->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1769058848 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4357.055628] LustreError: 6507:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4381.635944] Lustre: lustre-MDT0000: Export ffff8b8ea7cfa000 already connecting from 192.168.204.49@tcp [ 4381.642930] Lustre: Skipped 9 previous similar messages [ 4397.067825] LustreError: 6507:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4397.078827] Lustre: 6507:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b8eba9fd880 x1854988242128256/t0(0) o38->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1769058893 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4401.597519] LustreError: 6505:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4408.528326] LustreError: 6505:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 4414.158267] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 00:15:28 (1769058928) [ 4422.856403] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4426.600503] Lustre: Failing over lustre-MDT0000 [ 4426.907896] Lustre: server umount lustre-MDT0000 complete [ 4439.218570] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4439.221149] LDISKFS-fs (dm-0): recovery complete [ 4439.226882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4439.424407] Lustre: *** cfs_fail_loc=712, val=0*** [ 4439.427839] LustreError: 42232:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8b8ebaec3800 x1854988260937216/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4439.527209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4439.530172] Lustre: Skipped 4 previous similar messages [ 4439.564747] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4439.565343] Lustre: lustre-MDT0000: Aborting client recovery [ 4439.567383] Lustre: Skipped 9 previous similar messages [ 4439.574217] LustreError: 116440:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4439.589515] Lustre: 116477:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4439.594267] Lustre: 116477:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 4439.598871] Lustre: 116477:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4439.607053] Lustre: 116477:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4439.611383] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4439.617718] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4439.627834] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4439.688627] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 4439.694725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 4443.952479] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4444.643385] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4444.681280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4444.694607] Lustre: Skipped 16 previous similar messages [ 4459.947108] Lustre: Failing over lustre-MDT0000 [ 4460.011964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4460.014842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4460.018884] Lustre: Skipped 2 previous similar messages [ 4460.046955] LustreError: Skipped 5 previous similar messages [ 4460.228241] Lustre: server umount lustre-MDT0000 complete [ 4479.152776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.172394] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4484.658890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 4484.658931] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 4492.287036] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4494.397288] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4503.471043] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 00:16:58 (1769059018) [ 4503.550410] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnecting [ 4503.561398] Lustre: Skipped 2 previous similar messages [ 4512.457476] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 00:17:07 (1769059027) [ 4513.777926] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4513.786900] LustreError: 6515:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8e83f19500 x1854988242190976/t0(0) o700->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:421/0 lens 264/248 e 0 to 0 dl 1769059041 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4533.793407] Lustre: Failing over lustre-MDT0000 [ 4533.959847] LustreError: 118620:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4533.969215] LustreError: 118620:0:(obd_class.h:479:obd_check_dev()) Skipped 27 previous similar messages [ 4534.280648] Lustre: server umount lustre-MDT0000 complete [ 4535.776524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.790776] Lustre: Skipped 14 previous similar messages [ 4545.467868] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.483296] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 75 previous similar messages [ 4553.314666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.785735] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4568.110711] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 4568.110900] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 4574.049137] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4575.663852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4586.699854] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 00:18:22 (1769059102) [ 4588.935705] Lustre: Failing over lustre-OST0000 [ 4589.043558] Lustre: server umount lustre-OST0000 complete [ 4607.166513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4613.685635] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4621.166609] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4622.908200] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4696.110883] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 00:20:11 (1769059211) [ 4703.986598] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4706.178649] Lustre: Failing over lustre-MDT0000 [ 4706.258726] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 4706.264541] Lustre: Skipped 1 previous similar message [ 4706.445905] Lustre: server umount lustre-MDT0000 complete [ 4725.715117] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769059226/real 1769059226] req@ffff8b8eba9e3100 x1854988261085568/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769059242 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4725.746951] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4725.753811] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4725.760541] LustreError: Skipped 3 previous similar messages [ 4730.321069] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4730.330695] LDISKFS-fs (dm-0): recovery complete [ 4730.340894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.978110] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4737.994129] Lustre: Skipped 3 previous similar messages [ 4741.209199] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4742.269773] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4742.278399] Lustre: Skipped 3 previous similar messages [ 4742.312311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4742.312606] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4814.433084] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 00:22:09 (1769059329) [ 4816.133972] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4816.137040] Lustre: Skipped 2 previous similar messages [ 4829.673603] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 00:22:25 (1769059345) [ 4833.237818] Lustre: Failing over lustre-MDT0000 [ 4833.639168] Lustre: server umount lustre-MDT0000 complete [ 4852.446605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4856.462816] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4858.354689] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4858.356894] LustreError: 123865:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8e9ff6b800 x1854988242319104/t0(0) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:53/0 lens 328/344 e 0 to 0 dl 1769059428 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4916.153461] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnected, waiting for 2 clients in recovery for 1:04 [ 4916.182726] Lustre: 123865:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4916.268667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 4916.273083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 4920.256439] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4921.852899] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4930.567752] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 00:24:06 (1769059446) [ 4932.656563] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4939.374388] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4940.970883] Lustre: Failing over lustre-MDT0000 [ 4941.279260] Lustre: server umount lustre-MDT0000 complete [ 4961.968450] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4961.971178] LDISKFS-fs (dm-0): recovery complete [ 4961.978363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.938539] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4976.691172] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 4976.700152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3137) [ 4982.050286] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4983.863296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4993.985327] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 00:25:09 (1769059509) [ 4995.339128] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5003.956625] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5005.572992] Lustre: Failing over lustre-MDT0000 [ 5005.854932] Lustre: server umount lustre-MDT0000 complete [ 5027.046927] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5027.049867] LDISKFS-fs (dm-0): recovery complete [ 5027.063886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5037.178387] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5038.679590] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 5038.693465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3169) [ 5044.304530] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5045.722323] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5053.521629] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 00:26:09 (1769059569) [ 5054.667866] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5063.535983] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5065.022446] Lustre: Failing over lustre-MDT0000 [ 5065.193313] Lustre: server umount lustre-MDT0000 complete [ 5085.617568] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5085.621374] LDISKFS-fs (dm-0): recovery complete [ 5085.632393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5096.697531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5096.707100] Lustre: Skipped 7 previous similar messages [ 5096.738144] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5096.752121] Lustre: Skipped 9 previous similar messages [ 5100.351421] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5102.078295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5102.082921] Lustre: Skipped 31 previous similar messages [ 5102.186345] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3169) [ 5102.188827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 5111.574583] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 00:27:06 (1769059626) [ 5113.781616] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5113.789988] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5113.799941] LustreError: 6509:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8eb08b2c50 x1854988242379008/t257698037777(0) o35->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:266/0 lens 392/456 e 0 to 0 dl 1769059641 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5115.995721] Lustre: Failing over lustre-MDT0000 [ 5116.272515] Lustre: server umount lustre-MDT0000 complete [ 5117.409607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5117.413887] LustreError: Skipped 4 previous similar messages [ 5133.919000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.499339] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8ebae89500 x1854988261316992/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5142.515865] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 5147.667885] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5148.203487] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 5148.205106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3233) [ 5148.206927] Lustre: 6509:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ebae89500 x1854988242379008/t257698037777(0) o35->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:300/0 lens 392/456 e 0 to 0 dl 1769059675 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5155.615328] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5157.718920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5165.339128] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 00:28:00 (1769059680) [ 5166.302623] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5166.310238] LustreError: 6506:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8ebbb41f80 x1854988242393984/t261993005072(0) o36->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:362/0 lens 504/448 e 0 to 0 dl 1769059737 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5175.111522] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5177.031584] Lustre: Failing over lustre-MDT0000 [ 5177.174045] LustreError: 132250:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5177.178934] LustreError: 132250:0:(obd_class.h:479:obd_check_dev()) Skipped 59 previous similar messages [ 5177.298863] Lustre: server umount lustre-MDT0000 complete [ 5178.849409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5178.866523] Lustre: Skipped 31 previous similar messages [ 5178.875124] LustreError: 7781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5178.896506] LustreError: 7781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 222 previous similar messages [ 5199.280528] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5199.282543] LDISKFS-fs (dm-0): recovery complete [ 5199.287809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.137569] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5211.161857] Lustre: 10015:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ebae0f480 x1854988242393984/t261993005072(0) o36->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:407/0 lens 504/2880 e 0 to 0 dl 1769059782 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5211.163444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3265) [ 5211.177942] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 5217.475885] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5218.788845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5226.090297] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 00:29:01 (1769059741) [ 5226.995582] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5227.000339] LustreError: 12132:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8d8b8a5f80 x1854988242410112/t266287972368(0) o36->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:423/0 lens 504/448 e 0 to 0 dl 1769059798 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5228.750019] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5234.920885] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5236.536968] Lustre: Failing over lustre-MDT0000 [ 5236.723613] Lustre: server umount lustre-MDT0000 complete [ 5257.297077] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5257.301840] LDISKFS-fs (dm-0): recovery complete [ 5257.315936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5261.185296] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5262.898035] Lustre: 6508:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8e9ffcd180 x1854988242410368/t266287972369(0) o35->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:415/0 lens 392/456 e 0 to 0 dl 1769059790 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5262.902136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 5262.910326] Lustre: 6508:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5262.911340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3265) [ 5271.696468] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 00:29:47 (1769059787) [ 5272.620297] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5272.628231] Lustre: Skipped 1 previous similar message [ 5272.632985] LustreError: 6505:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8d88498e00 x1854988242423680/t270582939664(0) o36->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:469/0 lens 504/448 e 0 to 0 dl 1769059844 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5272.664016] LustreError: 6505:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 5274.374414] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5274.376541] Lustre: Skipped 1 previous similar message [ 5281.613762] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5283.233809] Lustre: Failing over lustre-MDT0000 [ 5283.301014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5283.304506] Lustre: Skipped 4 previous similar messages [ 5283.393340] Lustre: server umount lustre-MDT0000 complete [ 5315.899599] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5315.901381] LDISKFS-fs (dm-0): recovery complete [ 5315.907691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5337.846087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3329) [ 5337.849505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 5337.947785] Lustre: 6506:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ea7c4d880 x1854988242423680/t270582939664(0) o36->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:534/0 lens 504/2880 e 0 to 0 dl 1769059909 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5338.714483] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5351.340905] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 00:31:06 (1769059866) [ 5352.742958] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5354.808125] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5354.809878] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5354.823198] LustreError: 6508:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8e8bd6c000 x1854988242440576/t274877906960(0) o35->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:507/0 lens 392/456 e 0 to 0 dl 1769059882 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5363.091030] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5364.863965] Lustre: Failing over lustre-MDT0000 [ 5365.005633] Lustre: server umount lustre-MDT0000 complete [ 5384.144207] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769059884/real 1769059884] req@ffff8b8e8bd6ed80 x1854988261449856/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769059900 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5384.162996] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 5384.171125] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5384.182543] LustreError: Skipped 8 previous similar messages [ 5393.140725] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5393.147096] LDISKFS-fs (dm-0): recovery complete [ 5393.165780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5394.336217] LustreError: 138465:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 5394.341765] LustreError: 138465:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8b8e9ff6bb80 x1854988261455744/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1769059910 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5394.368812] LustreError: 138465:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 5395.915761] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5395.925380] Lustre: Skipped 8 previous similar messages [ 5399.677498] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5400.059118] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5400.068461] Lustre: Skipped 8 previous similar messages [ 5400.127361] Lustre: 6508:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8e9ff6b480 x1854988242440576/t274877906960(0) o35->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:552/0 lens 392/456 e 0 to 0 dl 1769059927 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5400.136770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 5400.139724] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 5413.019574] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 00:32:08 (1769059928) [ 5414.108551] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 5414.114973] LustreError: 12132:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8ea7467800 x1854988242453120/t279172874255(0) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:610/0 lens 664/608 e 0 to 0 dl 1769059985 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5473.217449] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnecting [ 5473.227822] Lustre: Skipped 1 previous similar message [ 5473.276780] Lustre: 6506:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ebae0c380 x1854988242453120/t279172874255(0) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:669/0 lens 664/3488 e 0 to 0 dl 1769060044 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5482.758870] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 00:33:17 (1769059997) [ 5491.135984] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5493.733133] Lustre: Failing over lustre-MDT0000 [ 5494.164864] Lustre: server umount lustre-MDT0000 complete [ 5517.329839] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5517.332213] LDISKFS-fs (dm-0): recovery complete [ 5517.339420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.040558] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5526.620602] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 5526.620880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3393) [ 5532.559137] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5533.793715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5552.656471] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 00:34:28 (1769060068) [ 5560.255707] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5562.232275] Lustre: Failing over lustre-MDT0000 [ 5562.339267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.346693] Lustre: Skipped 3 previous similar messages [ 5562.498502] Lustre: server umount lustre-MDT0000 complete [ 5586.051568] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5586.053162] LDISKFS-fs (dm-0): recovery complete [ 5586.060209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.108416] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795d4414b [ 5599.821290] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5600.880919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 5600.885726] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3393) [ 5607.666910] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5609.062881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5614.370681] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5627.582541] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 00:35:43 (1769060143) [ 5667.018586] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5668.509346] Lustre: Failing over lustre-MDT0000 [ 5669.132221] Lustre: server umount lustre-MDT0000 complete [ 5690.230148] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5690.237024] LDISKFS-fs (dm-0): recovery complete [ 5690.247343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5699.041770] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8d8aa25f80 x1854988261625088/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5699.068047] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 35 previous similar messages [ 5699.381397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.388704] Lustre: Skipped 7 previous similar messages [ 5699.449762] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5699.454411] Lustre: Skipped 7 previous similar messages [ 5703.632739] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5704.687669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5704.696896] Lustre: Skipped 32 previous similar messages [ 5705.019978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 5705.020646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 5711.571497] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5713.189420] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5762.751343] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 00:37:58 (1769060278) [ 5770.379703] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5772.109909] Lustre: Failing over lustre-MDT0000 [ 5772.537413] Lustre: server umount lustre-MDT0000 complete [ 5779.907810] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5779.937663] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 252 previous similar messages [ 5794.629247] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5794.631350] LDISKFS-fs (dm-0): recovery complete [ 5794.646388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5806.591575] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5807.702169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 5807.710880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 5813.461813] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5814.964557] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5824.351313] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5825.944484] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5831.638870] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 00:39:07 (1769060347) [ 5839.574014] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5901.635624] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5901.643215] Lustre: Skipped 1 previous similar message [ 5901.653840] LustreError: 114195:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8ebb8bbb80 x1854988245260160/t296352743435(0) o36->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:343/0 lens 66040/440 e 0 to 0 dl 1769060473 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5961.682996] Lustre: 6505:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8e82ef8850 x1854988245260160/t296352743435(0) o36->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:403/0 lens 66040/440 e 0 to 0 dl 1769060533 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5973.261516] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5974.811674] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 00:41:30 (1769060490) [ 5986.609626] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5989.022610] Lustre: Failing over lustre-MDT0000 [ 5989.155393] LustreError: 148976:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5989.161306] LustreError: 148976:0:(obd_class.h:479:obd_check_dev()) Skipped 63 previous similar messages [ 5989.226445] Lustre: server umount lustre-MDT0000 complete [ 5991.905832] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5991.933231] Lustre: Skipped 34 previous similar messages [ 6008.288191] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769060508/real 1769060508] req@ffff8b8ebaf1a300 x1854988262438272/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769060524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6008.351790] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6008.378722] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6008.391765] LustreError: Skipped 4 previous similar messages [ 6011.728976] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6011.731957] LDISKFS-fs (dm-0): recovery complete [ 6011.739846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6017.511566] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795d87e52 [ 6019.009730] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6019.015809] Lustre: Skipped 4 previous similar messages [ 6022.669899] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6023.968692] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6023.977775] Lustre: Skipped 4 previous similar messages [ 6024.003803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 6024.007669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 6029.853064] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6031.890361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6043.058587] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 00:42:38 (1769060558) [ 6062.058123] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6073.204241] Lustre: Failing over lustre-OST0000 [ 6073.272657] Lustre: server umount lustre-OST0000 complete [ 6073.313920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6073.320539] LustreError: Skipped 3 previous similar messages [ 6093.488620] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6093.491887] LDISKFS-fs (dm-2): recovery complete [ 6093.499485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6098.898851] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6113.290377] Lustre: Failing over lustre-OST0000 [ 6113.384963] Lustre: server umount lustre-OST0000 complete [ 6130.612953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6136.958630] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6143.225486] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6144.598321] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6182.450414] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 00:44:58 (1769060698) [ 6184.565390] Lustre: Failing over lustre-MDT0000 [ 6184.762422] Lustre: server umount lustre-MDT0000 complete [ 6202.189624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6207.778814] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6208.129377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 6208.130059] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 6228.090477] Lustre: Failing over lustre-MDT0000 [ 6228.284828] Lustre: server umount lustre-MDT0000 complete [ 6253.642342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6260.215373] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795d99040 [ 6265.671623] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6265.938082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 6265.943411] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 6274.290623] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6276.104777] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6284.373781] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 00:46:40 (1769060800) [ 6301.258317] Lustre: Failing over lustre-OST0000 [ 6301.480829] Lustre: server umount lustre-OST0000 complete [ 6322.602403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6322.907066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6322.912798] Lustre: Skipped 6 previous similar messages [ 6322.931759] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6322.943728] Lustre: Skipped 6 previous similar messages [ 6324.347315] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6324.358161] Lustre: Skipped 25 previous similar messages [ 6329.633568] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6336.842601] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6338.694634] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6349.543384] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 00:47:44 (1769060864) [ 6351.737552] Lustre: Failing over lustre-MDT0000 [ 6352.001407] Lustre: server umount lustre-MDT0000 complete [ 6367.966377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.026124] Lustre: *** cfs_fail_loc=605, val=0*** [ 6368.028782] LustreError: 157857:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0874ca0 failed: rc = -95 [ 6368.037907] LustreError: 157857:0:(obd_config.c:859:class_setup()) setup MGS failed (-95) [ 6368.042051] LustreError: 157857:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 6368.046564] LustreError: 157857:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 6368.049361] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 6368.052765] LustreError: 157857:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 6368.060774] Lustre: server umount lustre-MDT0000 complete [ 6368.063793] LustreError: 157857:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 6374.952706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6385.119885] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6385.208701] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 6385.211549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 6394.290837] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 00:48:29 (1769060909) [ 6401.945706] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6405.169595] Lustre: Failing over lustre-MDT0000 [ 6405.408517] Lustre: server umount lustre-MDT0000 complete [ 6405.607591] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6405.636813] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 184 previous similar messages [ 6431.295438] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6431.302672] LDISKFS-fs (dm-0): recovery complete [ 6431.315798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.350064] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8ea7d29500 x1854988262810624/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6438.916855] Lustre: *** cfs_fail_loc=707, val=0*** [ 6443.039794] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6498.267278] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 6498.329678] Lustre: 160024:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 6498.350373] Lustre: 160024:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 6498.831804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 6498.833400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 6503.927710] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6505.570780] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6517.624832] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 00:50:32 (1769061032) [ 6549.168791] LustreError: 12132:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b8ebb7ca680 x1854988246163968/t0(0) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:191/0 lens 664/0 e 0 to 0 dl 1769061076 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6549.195295] LustreError: 12132:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 6560.211610] LustreError: 12132:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6560.249937] LustreError: 6508:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b8ebb8b9180 x1854988246165248/t0(0) o35->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:202/0 lens 392/0 e 0 to 0 dl 1769061087 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6563.103449] LustreError: 12132:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b8d8742df80 x1854988246171648/t0(0) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:245/0 lens 576/0 e 0 to 0 dl 1769061130 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 6563.145224] LustreError: 12132:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 6565.352633] LustreError: 42228:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b8ea7c4f100 x1854988262872448/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:207/0 lens 544/0 e 0 to 0 dl 1769061092 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 6565.382364] LustreError: 42228:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 24 previous similar messages [ 6570.992922] LustreError: 42228:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b8ea7c4b100 x1854988262876544/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:213/0 lens 544/0 e 0 to 0 dl 1769061098 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 6571.030803] LustreError: 42228:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 3 previous similar messages [ 6580.126807] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 00:51:35 (1769061095) [ 6612.260465] LustreError: 41161:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 6623.264294] LustreError: 41161:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 6636.386299] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 00:52:31 (1769061151) [ 6665.950916] LustreError: 6507:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b8ea7465500 x1854988246193152/t0(0) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:308/0 lens 576/0 e 0 to 0 dl 1769061193 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6665.988697] LustreError: 6507:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 6671.000182] LustreError: 6507:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6683.968170] LustreError: 6505:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6683.987141] LustreError: 6505:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b8d8b8a7100 x1854988246215680/t0(0) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:365/0 lens 664/0 e 0 to 0 dl 1769061250 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6684.028312] LustreError: 6505:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 120 previous similar messages [ 6703.941161] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 00:53:39 (1769061219) [ 6799.521336] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 00:55:14 (1769061314) [ 6829.499163] LustreError: 10015:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b8ea7d28e00 x1854988246275968/t0(0) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:472/0 lens 576/0 e 0 to 0 dl 1769061357 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6829.525928] LustreError: 10015:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 6829.534845] LustreError: 10015:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6829.541921] LustreError: 10015:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 6829.970236] LustreError: 10015:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6845.927747] LustreError: 6508:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6845.944838] LustreError: 6508:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 6846.368794] LustreError: 6508:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6846.382120] LustreError: 6508:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 6880.488731] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 00:56:35 (1769061395) [ 6912.549471] Lustre: DEBUG MARKER: phase 2 [ 6920.511757] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 00:57:16 (1769061436) [ 7002.963279] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 00:58:38 (1769061518) [ 7004.490630] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 7006.499304] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 00:58:41 (1769061521) [ 7010.948780] Lustre: DEBUG MARKER: Started rundbench load pid=134514 ... [ 7019.076317] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7021.321435] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 7022.923059] Lustre: Failing over lustre-MDT0000 [ 7023.164476] LustreError: 166993:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7023.168923] LustreError: 166993:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 7023.357631] Lustre: server umount lustre-MDT0000 complete [ 7026.148479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7026.155021] LustreError: Skipped 2 previous similar messages [ 7026.160305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7026.170932] Lustre: Skipped 22 previous similar messages [ 7026.179970] LustreError: 6511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7026.186965] LustreError: 6511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 7043.045985] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769061543/real 1769061543] req@ffff8b8ea7465c00 x1854988263150720/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769061559 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7043.080492] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 7043.085890] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7043.104490] LustreError: Skipped 4 previous similar messages [ 7044.537516] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 7044.541207] LDISKFS-fs (dm-0): recovery complete [ 7044.549730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7053.289615] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8ebe7e4e00 x1854988263309824/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7053.556144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7053.566603] Lustre: Skipped 2 previous similar messages [ 7053.625689] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7053.633969] Lustre: Skipped 2 previous similar messages [ 7055.020594] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7055.024562] Lustre: Skipped 7 previous similar messages [ 7057.094246] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7058.923021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7058.929110] Lustre: Skipped 9 previous similar messages [ 7058.959514] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7058.966320] Lustre: Skipped 7 previous similar messages [ 7059.002773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 7059.011440] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 7063.437387] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7064.620475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7074.058464] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7076.373638] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 7077.914314] Lustre: Failing over lustre-MDT0001 [ 7077.946531] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 7078.188581] Lustre: server umount lustre-MDT0001 complete [ 7098.925341] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 7098.927212] LDISKFS-fs (dm-1): recovery complete [ 7098.936191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7102.575649] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7104.894165] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:589 to 0x280000400:609) [ 7104.895328] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:571 to 0x2c0000400:609) [ 7108.772359] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7110.039964] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7120.002734] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7122.442926] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 7124.129057] Lustre: Failing over lustre-MDT0000 [ 7124.405427] Lustre: server umount lustre-MDT0000 complete [ 7143.532587] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7143.537039] LDISKFS-fs (dm-0): recovery complete [ 7143.546599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7147.333298] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7149.078083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 7149.083128] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 7153.309225] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7154.641120] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7161.702037] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 01:01:17 (1769061677) [ 7288.454621] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7300.502145] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 7301.842656] Lustre: Failing over lustre-MDT0000 [ 7301.888300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7302.066950] Lustre: server umount lustre-MDT0000 complete [ 7321.518504] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7321.520228] LDISKFS-fs (dm-0): recovery complete [ 7321.529991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7332.176035] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7334.918843] Lustre: 172969:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 7334.938654] Lustre: 172969:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 52 previous similar messages [ 7335.423941] Lustre: 172969:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 7335.432520] Lustre: 172969:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 66 previous similar messages [ 7336.424121] Lustre: 172969:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 7336.435933] Lustre: 172969:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 278 previous similar messages [ 7337.315474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5517 to 0x280000401:5537) [ 7337.316255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5485 to 0x2c0000401:5505) [ 7340.055270] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7341.203489] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7393.345453] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 01:05:09 (1769061909) [ 7520.804444] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7532.868480] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 7534.444226] Lustre: Failing over lustre-MDT0001 [ 7534.472426] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 7534.796742] Lustre: server umount lustre-MDT0001 complete [ 7554.858723] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7554.860983] LDISKFS-fs (dm-1): recovery complete [ 7554.874763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7558.826871] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7561.619437] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1322 to 0x2c0000400:1345) [ 7561.619558] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1322 to 0x280000400:1345) [ 7565.152915] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7566.494965] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7694.131139] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7705.934798] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 7707.401887] Lustre: Failing over lustre-MDT0000 [ 7707.457483] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 7707.461588] Lustre: Skipped 3 previous similar messages [ 7708.641822] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7708.642696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7708.650359] Lustre: Skipped 20 previous similar messages [ 7708.659878] LustreError: Skipped 3 previous similar messages [ 7712.874313] LustreError: 176000:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7712.883537] LustreError: 176000:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 7712.997992] Lustre: server umount lustre-MDT0000 complete [ 7713.769752] LustreError: 10015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7713.787373] LustreError: 10015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 120 previous similar messages [ 7730.144251] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769062230/real 1769062230] req@ffff8b8ea7bec000 x1854988270820864/t0(0) o400->MGC192.168.204.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769062246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7730.168439] Lustre: 3655:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7730.176598] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7730.182813] LustreError: Skipped 2 previous similar messages [ 7732.866885] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7732.868625] LDISKFS-fs (dm-0): recovery complete [ 7732.876907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.396337] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795edc3b1 [ 7740.417717] Lustre: MGC192.168.204.149@tcp: Connection restored to 0@lo (at 0@lo) [ 7740.423099] Lustre: Skipped 17 previous similar messages [ 7740.619417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7740.623608] Lustre: Skipped 4 previous similar messages [ 7740.660343] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7740.664539] Lustre: Skipped 4 previous similar messages [ 7741.122520] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7741.130628] Lustre: Skipped 4 previous similar messages [ 7744.158518] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7748.594612] Lustre: lustre-MDT0000: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 7748.598657] Lustre: Skipped 4 previous similar messages [ 7748.657192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5909 to 0x280000401:5953) [ 7748.658248] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5876 to 0x2c0000401:5921) [ 7751.099359] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7752.635229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7758.888587] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 01:11:14 (1769062274) [ 7886.756820] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7898.452771] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 7900.061107] Lustre: Failing over lustre-MDT0001 [ 7900.096610] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 7900.099298] Lustre: Skipped 5 previous similar messages [ 7900.295423] Lustre: server umount lustre-MDT0001 complete [ 7920.270666] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7920.272808] LDISKFS-fs (dm-1): recovery complete [ 7920.281252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7924.080364] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7925.833062] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1498 to 0x280000400:1537) [ 7925.835854] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1498 to 0x2c0000400:1537) [ 7931.063101] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7932.638535] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8062.132423] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8074.267987] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 8075.793940] Lustre: Failing over lustre-MDT0000 [ 8075.846398] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.49@tcp (stopping) [ 8075.848717] Lustre: Skipped 1 previous similar message [ 8076.055556] Lustre: server umount lustre-MDT0000 complete [ 8097.347749] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8097.350341] LDISKFS-fs (dm-0): recovery complete [ 8097.356598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.279315] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8116.096796] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7050 to 0x2c0000401:7073) [ 8116.099751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7082 to 0x280000401:7105) [ 8118.980718] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8120.503559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8128.583732] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 01:17:23 (1769062643) [ 8138.857590] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8141.230930] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 8143.075032] Lustre: Failing over lustre-OST0000 [ 8143.170995] Lustre: server umount lustre-OST0000 complete [ 8163.941131] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8163.942889] LDISKFS-fs (dm-2): recovery complete [ 8163.954830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8168.735399] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8175.935981] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8177.660671] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8191.211335] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8193.560667] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 8194.949168] Lustre: Failing over lustre-OST0000 [ 8195.004716] Lustre: server umount lustre-OST0000 complete [ 8214.911510] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8214.913989] LDISKFS-fs (dm-2): recovery complete [ 8214.934268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8219.202681] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8225.781379] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8227.195585] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8236.072705] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 01:19:11 (1769062751) [ 8362.087317] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8367.834375] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8379.339101] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 8380.561259] Lustre: Failing over lustre-MDT0001 [ 8380.573646] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 8380.581035] LustreError: Skipped 3 previous similar messages [ 8380.583755] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.591718] Lustre: Skipped 11 previous similar messages [ 8380.594458] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8380.685504] LustreError: 185705:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8380.691790] LustreError: 185705:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 8380.851521] Lustre: server umount lustre-MDT0001 complete [ 8382.413851] LustreError: 114195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8382.437205] LustreError: 114195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 101 previous similar messages [ 8383.409626] Lustre: Failing over lustre-MDT0000 [ 8389.363873] Lustre: server umount lustre-MDT0000 complete [ 8408.080846] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8408.085671] LDISKFS-fs (dm-0): recovery complete [ 8408.096437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8408.111445] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8408.114504] LDISKFS-fs (dm-1): recovery complete [ 8408.126174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8408.237627] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8408.247646] LustreError: Skipped 1 previous similar message [ 8408.451514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8408.455569] Lustre: Skipped 4 previous similar messages [ 8408.477481] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8408.481726] Lustre: Skipped 4 previous similar messages [ 8411.183678] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8411.387420] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8413.114918] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8413.118895] Lustre: Skipped 4 previous similar messages [ 8414.098323] LustreError: 171706:0:(ldlm_lockd.c:914:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-lustre-MDT0001_UUID lock: ffff8b8e9fe3de00/0xff207ff795fbd833 lrc: 3/0,0 mode: PW/PW res: [0x240001f59:0x492:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x40000000000020 nid: 0@lo remote: 0xff207ff795fbd82c expref: 41 pid: 171706 timeout: 0 lvb_type: 0 [ 8414.650064] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8414.653707] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8414.659188] Lustre: Skipped 4 previous similar messages [ 8414.661896] Lustre: Skipped 15 previous similar messages [ 8414.704989] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1539 to 0x280000400:1569) [ 8414.705669] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1539 to 0x2c0000400:1569) [ 8419.554826] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7106 to 0x2c0000401:7137) [ 8419.555451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7169) [ 8421.808657] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8422.856163] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8423.933862] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8430.334956] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 01:22:26 (1769062946) [ 8434.858253] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8436.440773] Lustre: Failing over lustre-MDT0000 [ 8436.648910] Lustre: server umount lustre-MDT0000 complete [ 8453.206808] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8453.208512] LDISKFS-fs (dm-0): recovery complete [ 8453.212859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8454.085056] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 8455.715684] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8469.458245] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 8469.492408] Lustre: 187018:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ebe7e5880 x1854988269146368/t339302416721(339302416721) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:602/0 lens 520/3488 e 0 to 0 dl 1769062997 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 8469.564730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7201) [ 8469.565620] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7139 to 0x2c0000401:7169) [ 8471.676212] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8472.539436] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8477.276245] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 01:23:13 (1769062993) [ 8481.478079] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8482.988268] Lustre: Failing over lustre-MDT0000 [ 8483.140275] Lustre: server umount lustre-MDT0000 complete [ 8499.078985] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8499.081057] LDISKFS-fs (dm-0): recovery complete [ 8499.084929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8500.166385] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 8500.169109] LustreError: 187015:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8ebaa02a00 x1854988269146368/t339302416721(339302416721) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:632/0 lens 520/664 e 0 to 0 dl 1769063027 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 8500.886129] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8516.545393] Lustre: lustre-MDT0000: Client 61c43a80-9ad2-4b8b-bb68-67fc6884d641 (at 192.168.204.49@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 8516.553905] Lustre: 187766:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ea7b51c00 x1854988269146368/t339302416721(339302416721) o101->61c43a80-9ad2-4b8b-bb68-67fc6884d641@192.168.204.49@tcp:649/0 lens 520/3488 e 0 to 0 dl 1769063044 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 8516.592801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7233) [ 8516.592816] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7139 to 0x2c0000401:7201) [ 8518.541482] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8519.395533] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8524.254316] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 01:24:00 (1769063040) [ 8525.675823] Lustre: Failing over lustre-OST0000 [ 8525.727831] Lustre: server umount lustre-OST0000 complete [ 8527.417716] Lustre: Failing over lustre-MDT0000 [ 8527.512503] Lustre: server umount lustre-MDT0000 complete [ 8541.535335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8543.473565] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8544.602292] Lustre: lustre-MDT0000: Denying connection for new client 6a0f870b-9808-4773-86cd-2e503ce94802 (at 192.168.204.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 8544.615038] Lustre: Skipped 13 previous similar messages [ 8546.811901] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7139 to 0x2c0000401:7233) [ 8552.445202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8553.662997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7265) [ 8555.095207] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8560.417383] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 01:24:36 (1769063076) [ 8560.990766] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8560.992840] LustreError: 187027:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8d88882d80 x1854988276778240/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:693/0 lens 1312/4320 e 0 to 0 dl 1769063088 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8564.440463] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8565.341981] Lustre: Failing over lustre-MDT0000 [ 8565.576695] Lustre: server umount lustre-MDT0000 complete [ 8576.480133] Lustre: 187040:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769063077/real 1769063077] req@ffff8b8d88882a00 x1854988276778240/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1769063093 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8576.492779] Lustre: 187040:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8581.601497] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8581.603062] LDISKFS-fs (dm-0): recovery complete [ 8581.606825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.594111] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8587.270547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7297) [ 8587.270548] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7265) [ 8589.072377] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8589.856273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8594.548107] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 01:25:10 (1769063110) [ 8595.049527] LustreError: 187028:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8ebe354700 x1854988276807680/t360777252876(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:727/0 lens 2520/4320 e 0 to 0 dl 1769063122 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8598.962487] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8602.847364] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8603.832872] Lustre: Failing over lustre-MDT0001 [ 8603.843191] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 8603.845712] Lustre: Skipped 6 previous similar messages [ 8609.866452] Lustre: server umount lustre-MDT0001 complete [ 8626.052533] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8626.055144] LDISKFS-fs (dm-1): recovery complete [ 8626.063272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8628.214319] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8631.305698] Lustre: 187018:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ebabadc00 x1854988271400704/t21474837627(0) o36->6a0f870b-9808-4773-86cd-2e503ce94802@192.168.204.49@tcp:34/0 lens 560/2880 e 0 to 0 dl 1769063184 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8631.306324] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1580 to 0x2c0000400:1601) [ 8631.307463] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1580 to 0x280000400:1601) [ 8632.955531] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8633.716322] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8638.468797] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 01:25:54 (1769063154) [ 8638.985499] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8638.989226] Lustre: Skipped 1 previous similar message [ 8642.591150] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8646.396683] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8647.370065] Lustre: Failing over lustre-MDT0000 [ 8647.571353] Lustre: server umount lustre-MDT0000 complete [ 8663.960341] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8663.962786] LDISKFS-fs (dm-0): recovery complete [ 8663.969623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8666.366982] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8669.835296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7297) [ 8669.836047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7329) [ 8671.738478] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8672.556208] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8674.929880] Lustre: Failing over lustre-MDT0001 [ 8675.089903] Lustre: server umount lustre-MDT0001 complete [ 8691.757895] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8691.760983] LDISKFS-fs (dm-1): recovery complete [ 8691.773471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8694.249214] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8697.349634] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1612 to 0x280000400:1633) [ 8697.349635] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1612 to 0x2c0000400:1633) [ 8698.973571] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8699.717711] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8704.730817] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 01:27:00 (1769063220) [ 8705.204481] LustreError: 187027:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8ebcb14000 x1854988276895744/t365072220176(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:82/0 lens 2520/4320 e 0 to 0 dl 1769063232 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8705.214728] LustreError: 187027:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 8711.282122] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8714.524154] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8715.392968] Lustre: Failing over lustre-MDT0000 [ 8715.638670] Lustre: server umount lustre-MDT0000 complete [ 8717.373538] LustreError: 6492:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769063233 with bad export cookie 18383834380276363736 [ 8717.374632] Lustre: Failing over lustre-MDT0001 [ 8717.379235] LustreError: 6492:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8717.396450] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 8717.399542] Lustre: Skipped 4 previous similar messages [ 8722.915928] LustreError: 202729:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b8ebc3bc700 x1854988276907008/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8722.929214] LustreError: 202729:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8723.071421] Lustre: server umount lustre-MDT0001 complete [ 8739.881475] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8739.884818] LDISKFS-fs (dm-1): recovery complete [ 8739.887987] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8739.889899] LDISKFS-fs (dm-0): recovery complete [ 8739.894812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8739.896186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8739.992574] LustreError: 203822:0:(llog.c:1643:llog_backup()) MGC192.168.204.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8739.996834] Lustre: 203822:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8741.984135] LustreError: 203830:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 8741.991684] LustreError: 203830:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8b8e9fe71880 x1854988276907136/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1769063258 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8742.013032] LustreError: 203830:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 8744.719790] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8744.782944] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8747.684426] Lustre: 203880:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ea7d16680 x1854988271471360/t30064771142(0) o36->6a0f870b-9808-4773-86cd-2e503ce94802@192.168.204.49@tcp:151/0 lens 560/2880 e 0 to 0 dl 1769063301 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8747.689175] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1644 to 0x2c0000400:1665) [ 8747.689539] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1644 to 0x280000400:1665) [ 8758.828485] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7329) [ 8758.828789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7361) [ 8760.894694] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8761.705563] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8762.451737] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8766.873604] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 01:28:03 (1769063283) [ 8767.346744] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8767.349153] Lustre: Skipped 1 previous similar message [ 8774.105617] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8775.006825] Lustre: Failing over lustre-MDT0000 [ 8775.235095] Lustre: server umount lustre-MDT0000 complete [ 8790.758624] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8790.760837] LDISKFS-fs (dm-0): recovery complete [ 8790.764110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8792.497439] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8796.155874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7393) [ 8796.155993] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7361) [ 8797.553241] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8798.197674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8822.713664] Lustre: lustre-MDT0001: Client 6a0f870b-9808-4773-86cd-2e503ce94802 (at 192.168.204.49@tcp) reconnecting [ 8822.718517] Lustre: Skipped 2 previous similar messages [ 8822.732959] Lustre: 203838:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ea7c49880 x1854988271505408/t34359738438(0) o36->6a0f870b-9808-4773-86cd-2e503ce94802@192.168.204.49@tcp:239/0 lens 560/2880 e 0 to 0 dl 1769063389 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8826.310336] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 01:29:02 (1769063342) [ 8830.380616] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8831.235880] Lustre: Failing over lustre-MDT0001 [ 8831.355559] Lustre: server umount lustre-MDT0001 complete [ 8846.277904] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8846.280491] LDISKFS-fs (dm-1): recovery complete [ 8846.289717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8848.106498] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8851.967211] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1686 to 0x2c0000400:1729) [ 8851.967349] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1686 to 0x280000400:1729) [ 8853.408499] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8854.190348] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8858.820580] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 01:29:35 (1769063375) [ 8859.269043] LustreError: 203880:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8ea78b3800 x1854988271570688/t38654705734(0) o36->6a0f870b-9808-4773-86cd-2e503ce94802@192.168.204.49@tcp:236/0 lens 560/448 e 0 to 0 dl 1769063386 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8859.281690] LustreError: 203880:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 2 previous similar messages [ 8865.484971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8868.390310] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8869.103090] Lustre: Failing over lustre-MDT0000 [ 8869.403585] Lustre: server umount lustre-MDT0000 complete [ 8884.665870] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8884.667715] LDISKFS-fs (dm-0): recovery complete [ 8884.671337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8886.468139] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8889.872348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7425) [ 8889.872424] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7393) [ 8891.329167] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8892.013321] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8894.240913] Lustre: Failing over lustre-MDT0001 [ 8894.337542] Lustre: server umount lustre-MDT0001 complete [ 8910.410673] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8910.412976] LDISKFS-fs (dm-1): recovery complete [ 8910.424845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8912.200048] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8915.959705] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1740 to 0x280000400:1761) [ 8915.959816] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1740 to 0x2c0000400:1761) [ 8917.425867] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8918.077461] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8922.568033] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 01:30:38 (1769063438) [ 8929.525316] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8933.151861] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8933.943642] Lustre: Failing over lustre-MDT0000 [ 8934.175494] Lustre: server umount lustre-MDT0000 complete [ 8935.853061] LustreError: 6491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769063452 with bad export cookie 18383834380276377540 [ 8935.854702] Lustre: Failing over lustre-MDT0001 [ 8935.858271] LustreError: 6491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8935.985290] Lustre: server umount lustre-MDT0001 complete [ 8951.653485] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8951.655521] LDISKFS-fs (dm-1): recovery complete [ 8951.663938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8951.759537] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8951.761220] LDISKFS-fs (dm-0): recovery complete [ 8951.767360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8951.806052] LustreError: 214448:0:(llog.c:1643:llog_backup()) MGC192.168.204.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8951.813804] Lustre: 214448:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8960.481286] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8ea7da3480 x1854988277087104/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8960.489349] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 8962.396697] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8962.440351] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8966.776574] Lustre: 214477:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8ebe1a4000 x1854988271608704/t42949673030(0) o36->6a0f870b-9808-4773-86cd-2e503ce94802@192.168.204.49@tcp:344/0 lens 560/2880 e 0 to 0 dl 1769063494 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8966.781108] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1772 to 0x2c0000400:1793) [ 8966.781630] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1772 to 0x280000400:1793) [ 8966.787805] Lustre: 214477:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages [ 8966.810291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7425) [ 8966.810309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7457) [ 8968.320219] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8969.208363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8970.065395] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8975.666111] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 01:31:31 (1769063491) [ 8979.621123] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8980.715288] Lustre: Failing over lustre-MDT0001 [ 8980.720643] LustreError: 216151:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0x17:0x0].0x0 (ffff8b8d90e88e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8980.735655] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.49@tcp (stopping) [ 8980.739391] Lustre: Skipped 5 previous similar messages [ 8981.474458] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8981.490242] Lustre: Skipped 55 previous similar messages [ 8987.210907] LustreError: 216151:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 8987.213591] LustreError: 216151:0:(obd_class.h:479:obd_check_dev()) Skipped 139 previous similar messages [ 8987.285284] Lustre: server umount lustre-MDT0001 complete [ 8987.578526] LustreError: 214479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8987.591641] LustreError: 214479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 195 previous similar messages [ 9003.465101] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9003.468985] LDISKFS-fs (dm-1): recovery complete [ 9003.490513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9006.201867] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9009.145321] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1804 to 0x2c0000400:1825) [ 9009.145322] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1804 to 0x280000400:1825) [ 9010.809632] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9011.644663] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9016.257912] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 01:32:12 (1769063532) [ 9021.225161] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9022.377424] Lustre: Failing over lustre-MDT0000 [ 9022.584104] Lustre: server umount lustre-MDT0000 complete [ 9024.484942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9024.487894] LustreError: Skipped 12 previous similar messages [ 9038.910361] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9038.912064] LDISKFS-fs (dm-0): recovery complete [ 9038.917847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.985600] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9038.990393] LustreError: Skipped 9 previous similar messages [ 9039.117161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9039.119559] Lustre: Skipped 18 previous similar messages [ 9039.147715] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9039.152815] Lustre: Skipped 18 previous similar messages [ 9040.830751] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9040.834066] Lustre: Skipped 18 previous similar messages [ 9041.091261] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9044.453966] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9044.457771] Lustre: Skipped 61 previous similar messages [ 9044.489240] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9044.493804] Lustre: Skipped 18 previous similar messages [ 9044.528479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7489) [ 9044.528769] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7457) [ 9046.707960] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9047.684531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9053.808752] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 01:32:49 (1769063569) [ 9054.568504] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 9054.571273] Lustre: Skipped 5 previous similar messages [ 9059.173513] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9063.577883] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9064.675067] Lustre: Failing over lustre-MDT0000 [ 9064.873066] Lustre: server umount lustre-MDT0000 complete [ 9081.205968] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9081.208082] LDISKFS-fs (dm-0): recovery complete [ 9081.212571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9083.736722] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9086.958857] Lustre: 220935:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 9086.963496] Lustre: 220935:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 256 previous similar messages [ 9087.015966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7489) [ 9087.017509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7521) [ 9088.919633] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9089.839642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9092.402770] Lustre: Failing over lustre-MDT0001 [ 9092.513784] Lustre: server umount lustre-MDT0001 complete [ 9107.656234] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9107.658150] LDISKFS-fs (dm-1): recovery complete [ 9107.666339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9109.608899] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9113.102762] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1804 to 0x280000400:1857) [ 9113.102761] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1804 to 0x2c0000400:1857) [ 9114.656175] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9115.359920] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9119.736019] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 01:33:55 (1769063635) [ 9120.244684] LustreError: 171706:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b8d891c1880 x1854988277230080/t390842023950(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:497/0 lens 1728/4320 e 0 to 0 dl 1769063647 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 9120.255538] LustreError: 171706:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 4 previous similar messages [ 9123.632864] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9126.989187] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9127.905514] Lustre: Failing over lustre-MDT0000 [ 9128.221288] Lustre: server umount lustre-MDT0000 complete [ 9129.852493] LustreError: 7429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769063646 with bad export cookie 18383834380276387998 [ 9129.852752] Lustre: Failing over lustre-MDT0001 [ 9129.859779] LustreError: 7429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9129.862638] LustreError: 223971:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0x1f:0x0].0xf7117594 (ffff8b8ebad81500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9129.878766] LustreError: 223971:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 1 previous similar message [ 9136.097772] LustreError: 223971:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b8ea7464700 x1854988277238016/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 9136.113499] LustreError: 223971:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 9136.250726] Lustre: server umount lustre-MDT0001 complete [ 9152.939520] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9152.943742] LDISKFS-fs (dm-0): recovery complete [ 9152.949602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9152.995107] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9152.997170] LDISKFS-fs (dm-1): recovery complete [ 9153.009401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9155.040211] LustreError: 225057:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9155.041494] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b8d92af2d80 x1854988277238784/t0(0) o250->MGC192.168.204.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9155.054359] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795fcda53 [ 9157.165618] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9157.344708] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9160.667758] Lustre: 225085:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b8d92757b80 x1854988271689728/t55834574850(0) o36->6a0f870b-9808-4773-86cd-2e503ce94802@192.168.204.49@tcp:563/0 lens 496/2888 e 0 to 0 dl 1769063713 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 9160.677522] Lustre: 225085:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 9160.680396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7553) [ 9160.680600] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7521) [ 9160.680800] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1804 to 0x280000400:1889) [ 9160.681312] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1804 to 0x2c0000400:1889) [ 9162.344831] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9163.068893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9163.797227] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9168.303484] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 01:34:44 (1769063684) [ 9172.507460] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9173.340754] Lustre: Failing over lustre-MDT0000 [ 9173.438241] Lustre: server umount lustre-MDT0000 complete [ 9184.193969] Lustre: lustre-MDT0001: Client 6a0f870b-9808-4773-86cd-2e503ce94802 (at 192.168.204.49@tcp) reconnecting [ 9184.204688] Lustre: Skipped 1 previous similar message [ 9189.374510] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9189.377114] LDISKFS-fs (dm-0): recovery complete [ 9189.383468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9191.332452] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9195.045611] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7553) [ 9195.045622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7585) [ 9196.813857] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9197.585658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9202.497679] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 01:35:18 (1769063718) [ 9206.677249] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9207.543600] Lustre: Failing over lustre-MDT0001 [ 9207.683350] Lustre: server umount lustre-MDT0001 complete [ 9223.283862] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9223.285692] LDISKFS-fs (dm-1): recovery complete [ 9223.290930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9225.022523] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9228.791971] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1804 to 0x2c0000400:1921) [ 9228.792142] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1804 to 0x280000400:1921) [ 9230.297855] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9231.001607] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9234.827818] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 01:35:51 (1769063751) [ 9238.659151] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9241.835470] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9242.684693] Lustre: Failing over lustre-MDT0000 [ 9242.840946] Lustre: server umount lustre-MDT0000 complete [ 9257.493404] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9257.495423] LDISKFS-fs (dm-0): recovery complete [ 9257.498349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.150282] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9263.126464] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7585) [ 9263.126623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7617) [ 9264.648960] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9265.410123] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9267.555642] Lustre: Failing over lustre-MDT0001 [ 9267.649052] Lustre: server umount lustre-MDT0001 complete [ 9282.880180] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9282.882207] LDISKFS-fs (dm-1): recovery complete [ 9282.893421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9284.812348] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9288.189401] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1804 to 0x280000400:1953) [ 9288.190705] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1804 to 0x2c0000400:1953) [ 9289.687744] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9290.447869] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9294.498666] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 01:36:50 (1769063810) [ 9298.354577] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9301.337587] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9302.035108] Lustre: Failing over lustre-MDT0000 [ 9302.274239] Lustre: server umount lustre-MDT0000 complete [ 9303.691780] LustreError: 7429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769063820 with bad export cookie 18383834380276395586 [ 9303.692868] Lustre: Failing over lustre-MDT0001 [ 9303.697051] LustreError: 7429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9303.800178] Lustre: server umount lustre-MDT0001 complete [ 9318.673109] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9318.675346] LDISKFS-fs (dm-1): recovery complete [ 9318.680279] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9318.682321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9318.685802] LDISKFS-fs (dm-0): recovery complete [ 9318.689304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9318.764160] LustreError: 235707:0:(llog.c:1643:llog_backup()) MGC192.168.204.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 9318.767214] Lustre: 235707:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 9325.024161] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769063825/real 1769063825] req@ffff8b8ea7416a00 x1854988277373952/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769063841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9325.034022] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [ 9329.121863] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff207ff795fcf813 [ 9330.887705] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9331.029787] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9335.418816] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1804 to 0x280000400:1985) [ 9335.419125] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1804 to 0x2c0000400:1985) [ 9335.452375] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7235 to 0x2c0000401:7617) [ 9335.452768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7203 to 0x280000401:7649) [ 9336.776075] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9337.377453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9338.001343] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9342.030430] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 01:37:38 (1769063858) [ 9342.759764] Lustre: 236954:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 6a0f870b-9808-4773-86cd-2e503ce94802 at adminstrative request [ 9348.203823] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 01:37:44 (1769063864) [ 9350.791917] Lustre: Failing over lustre-MDT0000 [ 9351.014202] Lustre: server umount lustre-MDT0000 complete [ 9364.254929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.951286] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9369.625717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7700 to 0x280000401:7745) [ 9369.625717] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7669 to 0x2c0000401:7713) [ 9370.937235] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9371.593597] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9375.342509] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 01:38:11 (1769063891) [ 9379.905562] Lustre: Failing over lustre-OST0000 [ 9379.969282] Lustre: server umount lustre-OST0000 complete [ 9393.500888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9396.170360] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9399.920677] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9400.607609] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9405.215745] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 01:38:41 (1769063921) [ 9407.378071] Lustre: Failing over lustre-MDT0000 [ 9407.391504] LustreError: 7429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769063923 with bad export cookie 18383834380276408207 [ 9407.400812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9407.406758] Lustre: Skipped 13 previous similar messages [ 9407.530454] Lustre: server umount lustre-MDT0000 complete [ 9410.590708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9412.240186] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9412.995550] Lustre: lustre-MDT0000: Denying connection for new client 0a68fcec-7b9b-4c13-8ea1-ad76df66bb5d (at 192.168.204.49@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 9416.183228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7700 to 0x280000401:7777) [ 9416.183475] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7669 to 0x2c0000401:7745) [ 9420.623992] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 01:38:56 (1769063936) [ 9424.093301] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9425.048371] Lustre: Failing over lustre-OST0000 [ 9425.093884] Lustre: server umount lustre-OST0000 complete [ 9439.351910] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9439.353478] LDISKFS-fs (dm-2): recovery complete [ 9439.357340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9441.427341] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9444.853840] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9445.485317] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9449.352249] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 01:39:25 (1769063965) [ 9452.604773] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9454.597257] Lustre: Failing over lustre-OST0000 [ 9454.634411] Lustre: server umount lustre-OST0000 complete [ 9469.156890] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9469.159256] LDISKFS-fs (dm-2): recovery complete [ 9469.164907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9470.596805] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.49@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:7779 extent [0-4194303]: client csum 7f390a60, server csum ef970b0e [ 9471.408701] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9474.930307] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9475.564552] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9479.546476] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 01:39:55 (1769063995) [ 9482.758291] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9485.670990] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9488.261098] Lustre: Failing over lustre-MDT0000 [ 9488.449560] Lustre: server umount lustre-MDT0000 complete [ 9489.941601] LustreError: 10256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769064006 with bad export cookie 18383834380276422445 [ 9489.944239] Lustre: Failing over lustre-OST0000 [ 9489.947956] LustreError: 10256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9489.989297] Lustre: server umount lustre-OST0000 complete [ 9504.830822] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9504.832448] LDISKFS-fs (dm-0): recovery complete [ 9504.839338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9517.054856] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9520.793577] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7669 to 0x2c0000401:7777) [ 9532.399653] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9532.401133] LDISKFS-fs (dm-2): recovery complete [ 9532.408438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9534.801450] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9541.312043] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 01:40:57 (1769064057) [ 9546.463947] Lustre: Failing over lustre-OST0000 [ 9546.527896] Lustre: server umount lustre-OST0000 complete [ 9547.919240] Lustre: Failing over lustre-MDT0000 [ 9548.243856] Lustre: server umount lustre-MDT0000 complete [ 9561.285970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9562.839718] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9566.709949] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7669 to 0x2c0000401:7809) [ 9571.067325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9573.268778] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9574.088574] Lustre: lustre-OST0000: Denying connection for new client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 9574.095303] Lustre: Skipped 1 previous similar message [ 9579.449137] Lustre: lustre-OST0000: Denying connection for new client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 9579.459854] Lustre: Skipped 1 previous similar message [ 9584.568774] Lustre: lustre-OST0000: Denying connection for new client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 9594.808553] Lustre: lustre-OST0000: Denying connection for new client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:47 [ 9594.814580] Lustre: Skipped 1 previous similar message [ 9615.289088] Lustre: lustre-OST0000: Denying connection for new client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:27 [ 9615.294672] Lustre: Skipped 3 previous similar messages [ 9642.500121] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9642.502383] Lustre: 250014:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 0a68fcec-7b9b-4c13-8ea1-ad76df66bb5d@ [ 9642.507667] Lustre: 250014:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 9642.510938] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9642.521345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7828 to 0x280000401:7849) [ 9647.765238] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [ 9652.230653] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 9654.420250] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 01:42:50 (1769064170) [ 9655.831318] Lustre: Failing over lustre-OST0001 [ 9655.859099] LustreError: 251249:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9655.862029] LustreError: 251249:0:(obd_class.h:479:obd_check_dev()) Skipped 147 previous similar messages [ 9655.893304] Lustre: server umount lustre-OST0001 complete [ 9656.260407] LustreError: 42242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9656.267922] LustreError: 42242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 258 previous similar messages [ 9658.336779] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9658.340058] LustreError: Skipped 16 previous similar messages [ 9658.341810] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9658.346128] Lustre: Skipped 61 previous similar messages [ 9668.975385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9669.021899] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9669.024969] Lustre: Skipped 19 previous similar messages [ 9669.028463] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [ 9669.030698] Lustre: Skipped 19 previous similar messages [ 9670.443091] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9670.445851] Lustre: Skipped 19 previous similar messages [ 9670.878459] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9670.980453] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9670.980466] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9670.980494] Lustre: Skipped 19 previous similar messages [ 9670.983535] Lustre: Skipped 62 previous similar messages [ 9674.855550] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 01:43:11 (1769064191) [ 9676.180620] Lustre: Failing over lustre-OST0000 [ 9676.225739] Lustre: server umount lustre-OST0000 complete [ 9689.126900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9691.032821] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9691.054564] LustreError: 252992:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9691.057424] LustreError: 252992:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 76 previous similar messages [ 9696.736174] Lustre: *** cfs_fail_loc=715, val=40*** [ 9706.424610] Lustre: lustre-OST0000: Client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp) reconnected, waiting for 3 clients in recovery for 1:25 [ 9707.488641] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [ 9707.492693] Lustre: Skipped 1 previous similar message [ 9712.608191] Lustre: *** cfs_fail_loc=715, val=40*** [ 9712.609924] Lustre: Skipped 2 previous similar messages [ 9713.632128] Lustre: *** cfs_fail_loc=715, val=40*** [ 9722.808963] Lustre: lustre-OST0000: Client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp) reconnected, waiting for 3 clients in recovery for 1:08 [ 9728.992182] Lustre: *** cfs_fail_loc=715, val=40*** [ 9728.993731] Lustre: Skipped 1 previous similar message [ 9731.096115] LustreError: 252992:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9731.099256] LustreError: 252992:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 76 previous similar messages [ 9732.407703] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9732.899620] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9736.241895] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 01:44:12 (1769064252) [ 9737.576286] Lustre: Failing over lustre-MDT0000 [ 9737.850472] Lustre: server umount lustre-MDT0000 complete [ 9750.750082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.785132] LustreError: MGC192.168.204.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9750.789060] LustreError: Skipped 9 previous similar messages [ 9752.255051] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9756.133046] LustreError: 254439:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 9759.712165] Lustre: *** cfs_fail_loc=715, val=80*** [ 9759.713525] Lustre: Skipped 2 previous similar messages [ 9769.912632] Lustre: lustre-MDT0000: Client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp) reconnected, waiting for 2 clients in recovery for 1:26 [ 9769.915962] Lustre: Skipped 2 previous similar messages [ 9771.488986] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9776.096265] Lustre: *** cfs_fail_loc=715, val=80*** [ 9776.098447] Lustre: Skipped 1 previous similar message [ 9785.272927] Lustre: lustre-MDT0000: Client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp) reconnected, waiting for 2 clients in recovery for 1:11 [ 9787.872500] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9794.016123] Lustre: *** cfs_fail_loc=715, val=80*** [ 9794.017963] Lustre: Skipped 2 previous similar messages [ 9801.656651] Lustre: lustre-MDT0000: Client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 9804.256601] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9818.040416] Lustre: lustre-MDT0000: Client f5447ef7-07dd-4bbd-a998-8d4eee22af2b (at 192.168.204.49@tcp) reconnected, waiting for 2 clients in recovery for 0:38 [ 9819.617496] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9836.000555] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9836.216125] LustreError: 254439:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9836.221323] Lustre: 254439:0:(ldlm_lib.c:2931:target_recovery_thread()) too long recovery - read logs [ 9836.224936] LustreError: dumping log to /tmp/lustre-log.1769064352.254439 [ 9836.281306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7863 to 0x280000401:7881) [ 9836.281361] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7822 to 0x2c0000401:7841) [ 9837.636167] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9838.176946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9842.019912] Lustre: DEBUG MARKER: == replay-single test complete, duration 9562 sec ======== 01:45:58 (1769064358) [ 9842.543498] Lustre: DEBUG MARKER: === replay-single: start cleanup 01:45:58 (1769064358) === [ 9845.111739] Lustre: DEBUG MARKER: === replay-single: finish cleanup 01:46:01 (1769064361) === [ 9845.779149] Lustre: Failing over lustre-MDT0000 [ 9845.953295] Lustre: server umount lustre-MDT0000 complete [ 9861.033968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9862.425437] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9866.232529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7863 to 0x280000401:7913) [ 9866.232542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7822 to 0x2c0000401:7873) [ 9867.402404] Lustre: DEBUG MARKER: oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9867.934743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9881.557823] Lustre: server umount lustre-MDT0000 complete [ 9884.454689] LustreError: 10256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769064400 with bad export cookie 18383834380276454995 [ 9884.459208] LustreError: 10256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9884.560225] Lustre: server umount lustre-MDT0001 complete [ 9898.049427] Lustre: server umount lustre-OST0000 complete [ 9911.194297] Lustre: server umount lustre-OST0001 complete [ 9916.573818] Lustre: DEBUG MARKER: oleg449-server.virtnet: executing unload_modules_local [ 9917.661776] Key type lgssc unregistered [ 9917.793662] LNet: 259023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9917.796569] LNetError: 259023:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9917.805957] LNet: Removed LNI 192.168.204.149@tcp [ 9918.158138] Key type .llcrypt unregistered [ 9918.159082] Key type ._llcrypt unregistered