[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 514257393 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003438] x2apic enabled [ 0.005005] Switched APIC routing to physical x2apic. [ 0.006000] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011133] LSM: Security Framework initializing [ 0.012058] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.024567] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029681] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032116] *** VALIDATE tmpfs *** [ 0.033487] *** VALIDATE proc *** [ 0.034282] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.036286] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038037] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044308] debug: unmapping init [mem 0xffffffffa1859000-0xffffffffa1860fff] [ 0.046431] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047743] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048019] ... version: 2 [ 0.048955] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052015] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054412] rcu: Hierarchical SRCU implementation. [ 0.056971] smp: Bringing up secondary CPUs ... [ 0.057660] x86: Booting SMP configuration: [ 0.058729] .... node #0, CPUs: #1 #2 #3 [ 0.071188] smp: Brought up 1 node, 4 CPUs [ 0.073012] smpboot: Max logical packages: 1 [ 0.074018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.109890] node 0 deferred pages initialised in 33ms [ 0.115009] devtmpfs: initialized [ 0.116233] x86/mm: Memory block size: 128MB [ 0.119564] gcov: version magic: 0x41383552 [ 0.121491] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.122073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123339] pinctrl core: initialized pinctrl subsystem [ 0.124209] [ 0.124912] ************************************************************* [ 0.125016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.126013] ** ** [ 0.127013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.128013] ** ** [ 0.129014] ** This means that this kernel is built to expose internal ** [ 0.130014] ** IOMMU data structures, which may compromise security on ** [ 0.131012] ** your system. ** [ 0.132012] ** ** [ 0.133013] ** If you see this message and you are not debugging the ** [ 0.134015] ** kernel, report this immediately to your vendor! ** [ 0.135014] ** ** [ 0.136013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.137013] ************************************************************* [ 0.138846] NET: Registered protocol family 16 [ 0.139655] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.140083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.141162] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.143052] cpuidle: using governor menu [ 0.145010] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.146450] PCI: Using configuration type 1 for base access [ 0.147326] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.156145] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.157021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.159036] cryptd: max_cpu_qlen set to 1000 [ 0.160761] ACPI: Added _OSI(Module Device) [ 0.162012] ACPI: Added _OSI(Processor Device) [ 0.163008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.164089] ACPI: Added _OSI(Processor Aggregator Device) [ 0.170229] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.179057] ACPI: Interpreter enabled [ 0.180195] ACPI: PM: (supports S0 S3 S4 S5) [ 0.182013] ACPI: Using IOAPIC for interrupt routing [ 0.184135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.188603] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.202591] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.206056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.209028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.214082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.220860] acpiphp: Slot [2] registered [ 0.223134] acpiphp: Slot [5] registered [ 0.225131] acpiphp: Slot [6] registered [ 0.227132] acpiphp: Slot [7] registered [ 0.229120] acpiphp: Slot [8] registered [ 0.230138] acpiphp: Slot [9] registered [ 0.232258] acpiphp: Slot [10] registered [ 0.234380] acpiphp: Slot [3] registered [ 0.236168] acpiphp: Slot [4] registered [ 0.239118] acpiphp: Slot [11] registered [ 0.241118] acpiphp: Slot [12] registered [ 0.242165] acpiphp: Slot [13] registered [ 0.243000] acpiphp: Slot [14] registered [ 0.244135] acpiphp: Slot [15] registered [ 0.247124] acpiphp: Slot [16] registered [ 0.249150] acpiphp: Slot [17] registered [ 0.251120] acpiphp: Slot [18] registered [ 0.253118] acpiphp: Slot [19] registered [ 0.255113] acpiphp: Slot [20] registered [ 0.256229] acpiphp: Slot [21] registered [ 0.258091] acpiphp: Slot [22] registered [ 0.259096] acpiphp: Slot [23] registered [ 0.261119] acpiphp: Slot [24] registered [ 0.263103] acpiphp: Slot [25] registered [ 0.264078] acpiphp: Slot [26] registered [ 0.265161] acpiphp: Slot [27] registered [ 0.267204] acpiphp: Slot [28] registered [ 0.269119] acpiphp: Slot [29] registered [ 0.271143] acpiphp: Slot [30] registered [ 0.273125] acpiphp: Slot [31] registered [ 0.275105] PCI host bridge to bus 0000:00 [ 0.277024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.281032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.291022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.298480] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.303067] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.308038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.319122] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.326619] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.345000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.353000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.363019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.368019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.373577] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376724] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391072] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 14648 usecs [ 0.403542] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.415014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.443019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.452016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.473356] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.492017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.507018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.550018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.581218] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.600017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.611021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.659015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.676000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.687018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.699018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.733016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.763186] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.775016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.792017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.821020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.834000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.840016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.846019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.867017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.883126] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.893016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.905018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.933017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.974035] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.984131] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.986436] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.988398] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.991204] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.994000] iommu: Default domain type: Passthrough [ 0.996328] SCSI subsystem initialized [ 0.998142] ACPI: bus type USB registered [ 1.004710] usbcore: registered new interface driver usbfs [ 1.009106] usbcore: registered new interface driver hub [ 1.012095] usbcore: registered new device driver usb [ 1.014216] pps_core: LinuxPPS API ver. 1 registered [ 1.015011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.017061] PTP clock support registered [ 1.019565] EDAC MC: Ver: 3.0.0 [ 1.020000] PCI: Using ACPI for IRQ routing [ 1.022111] NetLabel: Initializing [ 1.024009] NetLabel: domain hash size = 128 [ 1.026010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.028191] NetLabel: unlabeled traffic allowed by default [ 1.031360] vgaarb: loaded [ 1.033087] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.035010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.042791] clocksource: Switched to clocksource kvm-clock [ 1.258175] VFS: Disk quotas dquot_6.6.0 [ 1.263623] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.271046] *** VALIDATE ramfs *** [ 1.274190] *** VALIDATE hugetlbfs *** [ 1.278719] pnp: PnP ACPI init [ 1.282787] pnp: PnP ACPI: found 6 devices [ 1.302662] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.307051] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.311648] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.316362] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.320515] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.322263] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.338218] NET: Registered protocol family 2 [ 1.340516] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.360531] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.367220] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.378996] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.391162] TCP: Hash tables configured (established 65536 bind 65536) [ 1.397588] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.403309] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.412114] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.420435] NET: Registered protocol family 1 [ 1.425625] RPC: Registered named UNIX socket transport module. [ 1.428674] RPC: Registered udp transport module. [ 1.432018] RPC: Registered tcp transport module. [ 1.434687] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.437949] NET: Registered protocol family 44 [ 1.439690] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.442273] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.444527] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.452695] PCI: CLS 0 bytes, default 64 [ 1.460173] Unpacking initramfs... [ 4.675977] debug: unmapping init [mem 0xffff97b1fcc54000-0xffff97b1fffbffff] [ 4.683595] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.686836] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.694038] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.877422] Initialise system trusted keyrings [ 6.880676] Key type blacklist registered [ 6.886579] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.910561] zbud: loaded [ 6.920131] *** VALIDATE nfs *** [ 6.922435] *** VALIDATE nfs4 *** [ 6.926421] pstore: using deflate compression [ 6.932692] Platform Keyring initialized [ 7.383549] NET: Registered protocol family 38 [ 7.385806] Key type asymmetric registered [ 7.388495] Asymmetric key parser 'x509' registered [ 7.391354] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.395513] io scheduler mq-deadline registered [ 7.397945] io scheduler kyber registered [ 7.400336] io scheduler bfq registered [ 7.402537] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.406221] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.410132] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.413395] ACPI: Power Button [PWRF] [ 7.420326] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.429670] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.451604] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.467420] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.495807] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.526693] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.565791] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.576478] Non-volatile memory driver v1.3 [ 7.580439] Linux agpgart interface v0.103 [ 7.663929] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 7.677782] vda: detected capacity change from 0 to 68624384 [ 7.749256] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.756687] vdb: detected capacity change from 0 to 1073741824 [ 7.821546] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.830418] vdc: detected capacity change from 0 to 2621440000 [ 7.879867] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.889530] vdd: detected capacity change from 0 to 2621440000 [ 7.928845] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.936180] vde: detected capacity change from 0 to 4294967296 [ 7.961352] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.967315] vdf: detected capacity change from 0 to 4294967296 [ 7.979450] libphy: Fixed MDIO Bus: probed [ 8.079338] usbcore: registered new interface driver usbserial_generic [ 8.083084] usbserial: USB Serial support registered for generic [ 8.087269] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.093938] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.096152] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.102680] mousedev: PS/2 mouse device common for all mice [ 8.105267] rtc_cmos 00:05: RTC can wake from S4 [ 8.110864] rtc_cmos 00:05: registered as rtc0 [ 8.112206] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.113991] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.114520] intel_pstate: CPU model not supported [ 8.123496] hid: raw HID events driver (C) Jiri Kosina [ 8.125133] usbcore: registered new interface driver usbhid [ 8.131029] usbhid: USB HID core driver [ 8.134207] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.140512] drop_monitor: Initializing network drop monitor service [ 8.145349] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.152432] Initializing XFRM netlink socket [ 8.154892] NET: Registered protocol family 10 [ 8.159553] Segment Routing with IPv6 [ 8.161919] NET: Registered protocol family 17 [ 8.168834] mpls_gso: MPLS GSO support [ 8.175551] RAS: Correctable Errors collector initialized. [ 8.177889] AVX version of gcm_enc/dec engaged. [ 8.180169] AES CTR mode by8 optimization enabled [ 8.333331] sched_clock: Marking stable (8333293792, 0)->(10009154113, -1675860321) [ 8.337372] registered taskstats version 1 [ 8.339742] Loading compiled-in X.509 certificates [ 8.341769] zswap: loaded using pool lzo/zbud [ 8.404826] Key type big_key registered [ 8.427741] Key type encrypted registered [ 8.429723] ima: No TPM chip found, activating TPM-bypass! [ 8.432353] ima: Allocated hash algorithm: sha1 [ 8.434241] ima: No architecture policies found [ 8.436399] evm: Initialising EVM extended attributes: [ 8.438940] evm: security.selinux [ 8.440597] evm: security.ima [ 8.441619] evm: security.capability [ 8.443480] evm: HMAC attrs: 0x1 [ 8.446587] rtc_cmos 00:05: setting system clock to 2026-01-22 10:47:46 UTC (1769078866) [ 8.461396] debug: unmapping init [mem 0xffffffffa2803000-0xffffffffa29fffff] [ 8.465886] debug: unmapping init [mem 0xffffffffa1582000-0xffffffffa1858fff] [ 8.475128] Write protecting the kernel read-only data: 28672k [ 8.479389] debug: unmapping init [mem 0xffffffff9fc03000-0xffffffff9fdfffff] [ 8.483411] debug: unmapping init [mem 0xffffffffa0514000-0xffffffffa05fffff] [ 8.631703] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.655150] systemd[1]: Detected virtualization kvm. [ 8.664806] systemd[1]: Detected architecture x86-64. [ 8.666555] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.752877] systemd[1]: No hostname configured. [ 8.756513] systemd[1]: Set hostname to . [ 8.759907] random: systemd: uninitialized urandom read (16 bytes read) [ 8.762910] systemd[1]: Initializing machine ID from random generator. [ 8.902362] random: ln: uninitialized urandom read (6 bytes read) [ 9.170763] random: systemd: uninitialized urandom read (16 bytes read) [ 9.175491] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 9.208787] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 9.223889] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.641957] device-mapper: uevent: version 1.0.3 [ 10.645515] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 12.098172] virtio_net virtio0 ens2: renamed from eth0 [ 12.174859] random: fast init done [ 12.497624] scsi host0: ata_piix [ 12.508941] scsi host1: ata_piix [ 12.513677] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.519123] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.955707] random: crng init done [ 16.958089] random: 7 urandom warning(s) missed due to ratelimiting [ 17.682919] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 18.912207] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ 19.995474] hrtimer: interrupt took 11535400 ns [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.263218] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.047850] SELinux: Disabled at runtime. [ 22.165654] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.179856] systemd[1]: Detected virtualization kvm. [ 22.181343] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.353831] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.367674] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.408230] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.421956] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.432951] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.466326] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.493495] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 24.948423] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.707596] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.764858] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.926125] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.929439] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.035478] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit)[ 38.642956] Key type dns_resolver registered [*** ] A start job is running for Configur…only root support (14s / no limit) [** ] A start job is running for Configur…only root support (15s / no limit) [* ] A start job is running for Configur…only root support (15s / no limit)[ 39.955227] NFS: Registering the id_resolver key type [ 39.966254] Key type id_resolver registered [ 39.974382] Key type id_legacy registered [** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg451-server login: [ 101.979836] libcfs: loading out-of-tree module taints kernel. [ 102.020745] Key type ._llcrypt registered [ 102.022904] Key type .llcrypt registered [ 102.104517] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_hostid [ 118.899211] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 121.197487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.208258] alg: No test for adler32 (adler32-zlib) [ 122.782651] Lustre: Lustre: Build Version: 2.17.50_3_gca5b8be [ 123.602618] LNet: Added LNI 192.168.204.151@tcp [8/256/0/180] [ 125.295320] Key type lgssc registered [ 127.152259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 147.441163] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 149.369889] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 159.183937] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.218522] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.157382] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 193.433820] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing load_modules_local [ 208.579652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.720978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.789194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.206528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 210.254552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 210.375622] Lustre: lustre-MDT0000: new disk, initializing [ 210.521799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.546956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 215.446942] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.763141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 229.860686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 230.015190] Lustre: 6526:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 230.068829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 230.075349] Lustre: Skipped 1 previous similar message [ 230.210620] Lustre: lustre-MDT0001: new disk, initializing [ 230.312725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 230.340861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 230.353574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 235.076337] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.028602] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 250.850345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 250.943758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 251.271977] Lustre: lustre-OST0000: new disk, initializing [ 251.277561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 251.373742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.724412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 253.740487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.893507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 258.356819] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.753735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 273.834552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 273.924452] Lustre: lustre-OST0001: new disk, initializing [ 273.931103] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 273.991933] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 280.072599] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.668357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 281.686117] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 281.740787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 292.106558] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 298.632795] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 309.478388] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.869744] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing yml_node [ 318.591279] Lustre: DEBUG MARKER: Client: 2.17.50.3 [ 321.099911] Lustre: DEBUG MARKER: MDS: 2.17.50.3 [ 323.792910] Lustre: DEBUG MARKER: OSS: 2.17.50.3 [ 325.638726] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Thu Jan 22 05:53:02 EST 2026 [ 343.620380] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 345.500252] Lustre: DEBUG MARKER: === replay-single: start setup 05:53:22 (1769079202) === [ 348.892441] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing check_config_client /mnt/lustre [ 371.124556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 374.936831] Lustre: 13213:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 379.683715] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 385.603415] Lustre: DEBUG MARKER: === replay-single: finish setup 05:54:02 (1769079242) === [ 387.483906] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 05:54:04 (1769079244) [ 396.360591] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 398.351193] Lustre: Failing over lustre-MDT0000 [ 398.533781] LustreError: 14085:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 398.640724] Lustre: server umount lustre-MDT0000 complete [ 399.336165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 399.338961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.456261] LustreError: 8443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 404.498266] LustreError: 8443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 405.103498] LustreError: 10309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 409.569435] LustreError: 6532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 409.588400] LustreError: 6532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 414.695004] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 414.706653] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 415.776144] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769079257/real 1769079257] req@ffff97b1434fb800 x1855013761699072/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769079273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 415.816436] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 420.834755] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 420.853191] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 422.136922] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 422.139967] LDISKFS-fs (dm-0): recovery complete [ 422.157025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 426.230390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.923448] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 431.022153] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 431.617152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 431.666162] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 439.002882] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 440.648528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 449.264257] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 05:55:06 (1769079306) [ 451.412910] Lustre: Failing over lustre-OST0000 [ 451.494305] LustreError: 15516:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 451.502183] LustreError: 15516:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 451.579794] Lustre: server umount lustre-OST0000 complete [ 452.072103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.074568] LustreError: 8418:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.093282] Lustre: Skipped 4 previous similar messages [ 452.121576] LustreError: 8418:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 470.070122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 470.293483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.014055] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 472.132885] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 472.135476] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 472.157603] Lustre: Skipped 3 previous similar messages [ 476.325334] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 484.196439] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 485.957550] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 495.838318] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 05:55:52 (1769079352) [ 503.130430] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 505.676677] Lustre: Failing over lustre-MDT0000 [ 505.800283] LustreError: 17223:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 505.804326] LustreError: 17223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 506.047217] Lustre: server umount lustre-MDT0000 complete [ 506.336894] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.337192] LustreError: 6538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.353222] Lustre: Skipped 3 previous similar messages [ 506.384282] LustreError: 6538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 522.193134] Lustre: 3666:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769079364/real 1769079364] req@ffff97b141e81c00 x1855013761761920/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769079380 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 522.237868] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 529.493195] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 529.495340] LDISKFS-fs (dm-0): recovery complete [ 529.514144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.697117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.299234] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 538.093264] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 538.101423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 538.108476] Lustre: Skipped 1 previous similar message [ 539.554823] Lustre: lustre-MDT0000: Denying connection for new client 010feece-badf-40c1-a60f-7d11ce16d8ce (at 192.168.204.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 544.896410] Lustre: lustre-MDT0000: Denying connection for new client 010feece-badf-40c1-a60f-7d11ce16d8ce (at 192.168.204.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 550.003392] Lustre: lustre-MDT0000: Denying connection for new client 010feece-badf-40c1-a60f-7d11ce16d8ce (at 192.168.204.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 555.117943] Lustre: lustre-MDT0000: Denying connection for new client 010feece-badf-40c1-a60f-7d11ce16d8ce (at 192.168.204.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 560.243865] Lustre: lustre-MDT0000: Denying connection for new client 010feece-badf-40c1-a60f-7d11ce16d8ce (at 192.168.204.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 570.478990] Lustre: lustre-MDT0000: Denying connection for new client 010feece-badf-40c1-a60f-7d11ce16d8ce (at 192.168.204.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 570.502258] Lustre: Skipped 1 previous similar message [ 590.957536] Lustre: lustre-MDT0000: Denying connection for new client 010feece-badf-40c1-a60f-7d11ce16d8ce (at 192.168.204.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:17 [ 590.987869] Lustre: Skipped 3 previous similar messages [ 608.504124] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 608.509800] Lustre: 17876:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 59a5e7c5-8b14-4bdf-8096-0f3fca328d9c@ [ 608.521827] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 608.536599] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 608.538189] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 608.550517] Lustre: Skipped 2 previous similar messages [ 608.599148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 608.600903] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 619.620358] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 05:57:56 (1769079476) [ 626.436554] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 628.137915] Lustre: Failing over lustre-MDT0000 [ 628.193069] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.200923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 628.202086] Lustre: Skipped 1 previous similar message [ 628.212221] Lustre: Skipped 3 previous similar messages [ 628.277319] LustreError: 18933:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 628.282301] LustreError: 18933:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 628.342028] Lustre: server umount lustre-MDT0000 complete [ 633.311942] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.330216] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 649.170645] Lustre: 3664:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769079491/real 1769079491] req@ffff97b14194ea00 x1855013761821824/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769079507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 649.208741] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 651.567498] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 651.569983] LDISKFS-fs (dm-0): recovery complete [ 651.581515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.441916] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc3f95abb2721792d [ 659.450329] Lustre: MGC192.168.204.151@tcp: Connection restored to 0@lo (at 0@lo) [ 659.807356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.854982] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 664.318310] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 665.057593] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 666.715665] Lustre: lustre-MDT0000: Denying connection for new client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 666.740562] Lustre: Skipped 3 previous similar messages [ 733.293654] Lustre: lustre-MDT0000: Denying connection for new client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:02 [ 733.316099] Lustre: Skipped 12 previous similar messages [ 735.500208] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 735.511588] Lustre: 19586:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 010feece-badf-40c1-a60f-7d11ce16d8ce@ [ 735.535041] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 735.557785] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 735.559574] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 735.580216] Lustre: Skipped 3 previous similar messages [ 735.626161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 735.626352] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 746.722230] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 06:00:03 (1769079603) [ 752.794299] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 754.389895] Lustre: Failing over lustre-MDT0000 [ 754.498735] LustreError: 20646:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 754.507628] LustreError: 20646:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 754.552811] Lustre: server umount lustre-MDT0000 complete [ 756.199627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 756.213101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.227355] Lustre: Skipped 2 previous similar messages [ 756.231677] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 756.260442] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 773.611481] Lustre: 3664:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769079615/real 1769079615] req@ffff97b27bfaed80 x1855013761877376/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769079631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 773.646397] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 775.341194] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 775.346940] LDISKFS-fs (dm-0): recovery complete [ 775.353545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.127263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.176592] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 785.518698] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 788.308798] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 789.478459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 789.534432] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 789.566835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 789.579186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 795.236058] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 796.901620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 805.156978] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 06:01:02 (1769079662) [ 813.507423] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 815.748691] Lustre: Failing over lustre-MDT0000 [ 815.936099] LustreError: 22488:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 815.951624] LustreError: 22488:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 816.129223] Lustre: server umount lustre-MDT0000 complete [ 820.192852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.199752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 820.208627] Lustre: Skipped 5 previous similar messages [ 836.575442] Lustre: 3664:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769079678/real 1769079678] req@ffff97b27bfadf80 x1855013761912320/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769079694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 836.610323] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 837.533711] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 837.536146] LDISKFS-fs (dm-0): recovery complete [ 837.547730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.059264] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 848.749843] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 851.549550] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 852.454043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 852.467323] Lustre: Skipped 3 previous similar messages [ 852.583537] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 852.630693] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 852.631950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 858.779213] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 860.609675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 869.463612] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 06:02:06 (1769079726) [ 877.357724] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 879.156855] Lustre: Failing over lustre-MDT0000 [ 879.326062] LustreError: 24315:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 879.333051] LustreError: 24315:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 879.458199] Lustre: server umount lustre-MDT0000 complete [ 883.168439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.289548] LustreError: 10309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.315374] LustreError: 10309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 76 previous similar messages [ 899.551476] Lustre: 3666:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769079741/real 1769079741] req@ffff97b27b826300 x1855013761947520/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769079757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 899.584827] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 901.489765] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 901.496349] LDISKFS-fs (dm-0): recovery complete [ 901.507461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.815897] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b27c194a80 x1855013761957120/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 911.150502] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 912.467164] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 915.341792] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 916.458693] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 916.469679] Lustre: Skipped 4 previous similar messages [ 916.621137] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 916.679775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:193) [ 916.683397] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 922.876184] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 924.585455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 931.998172] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 06:03:09 (1769079789) [ 938.589566] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 940.229928] Lustre: Failing over lustre-MDT0000 [ 940.327678] LustreError: 26143:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 940.330811] LustreError: 26143:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 940.417596] Lustre: server umount lustre-MDT0000 complete [ 942.050102] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.062722] Lustre: Skipped 5 previous similar messages [ 958.431869] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769079800/real 1769079800] req@ffff97b2777e6680 x1855013761982720/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769079816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 958.449921] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 961.138772] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 961.141452] LDISKFS-fs (dm-0): recovery complete [ 961.146546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.672579] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b26f575880 x1855013761991168/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 968.874877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.887495] Lustre: Skipped 2 previous similar messages [ 968.920262] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 969.850260] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 972.830788] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 974.371171] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 974.409322] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 974.409537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 980.754724] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 982.554391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 991.059846] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 06:04:08 (1769079848) [ 997.913962] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 999.831241] Lustre: Failing over lustre-MDT0000 [ 999.906910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.916751] Lustre: Skipped 3 previous similar messages [ 1002.137789] Lustre: server umount lustre-MDT0000 complete [ 1021.407346] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769079863/real 1769079863] req@ffff97b27b8ea680 x1855013762017408/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769079879 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1021.453635] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1024.084217] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1024.087200] LDISKFS-fs (dm-0): recovery complete [ 1024.093565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.673584] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc3f95abb2721926a [ 1031.954431] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1036.495212] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1037.440919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1037.442938] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1043.486319] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1044.718643] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1051.650809] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 06:05:08 (1769079908) [ 1052.772927] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1052.778802] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1052.780866] LustreError: 22398:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b142fc2a00 x1855013743903488/t38654705666(0) o35->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:161/0 lens 392/456 e 0 to 0 dl 1769079921 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1061.470187] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1063.579730] Lustre: Failing over lustre-MDT0000 [ 1063.720593] LustreError: 29847:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1063.725586] LustreError: 29847:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 1063.931268] Lustre: server umount lustre-MDT0000 complete [ 1068.000180] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1068.002097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1068.018727] Lustre: Skipped 7 previous similar messages [ 1085.763100] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1085.768581] LDISKFS-fs (dm-0): recovery complete [ 1085.780103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1094.623580] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b270111f80 x1855013762063104/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1094.885757] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1096.062433] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1096.067492] Lustre: Skipped 1 previous similar message [ 1099.683795] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1100.267858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1100.278172] Lustre: Skipped 11 previous similar messages [ 1100.329138] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1100.342181] Lustre: 22398:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b270304700 x1855013743903488/t38654705666(0) o35->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:209/0 lens 392/456 e 0 to 0 dl 1769079969 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1100.343286] Lustre: Skipped 1 previous similar message [ 1100.419587] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1100.420608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1106.563994] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1108.192410] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1115.447640] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 06:06:12 (1769079972) [ 1122.490326] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1124.318090] Lustre: Failing over lustre-MDT0000 [ 1124.506977] Lustre: server umount lustre-MDT0000 complete [ 1125.855688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1142.240472] Lustre: 3664:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769079983/real 1769079983] req@ffff97b274ae9c00 x1855013762086912/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769079999 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1142.258686] Lustre: 3664:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1142.263658] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1142.270241] LustreError: Skipped 1 previous similar message [ 1145.046437] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1145.049469] LDISKFS-fs (dm-0): recovery complete [ 1145.057768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1147.363592] LustreError: 7474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.390434] LustreError: 7474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 162 previous similar messages [ 1152.484784] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc3f95abb27219bf5 [ 1152.738938] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1156.755945] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1158.261305] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1158.266880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1163.945759] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1165.538550] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1173.463695] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 06:07:10 (1769080030) [ 1180.479334] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1181.586833] Lustre: *** cfs_fail_loc=114, val=0*** [ 1184.083191] Lustre: Failing over lustre-MDT0000 [ 1184.426211] Lustre: server umount lustre-MDT0000 complete [ 1188.833262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1206.269253] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1206.274343] LDISKFS-fs (dm-0): recovery complete [ 1206.285193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.460700] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc3f95abb2721a0f6 [ 1220.379102] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1221.225722] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1221.226306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1227.516864] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1229.151774] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1237.265728] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 06:08:14 (1769080094) [ 1243.820853] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1244.621257] Lustre: *** cfs_fail_loc=128, val=0*** [ 1247.171171] Lustre: Failing over lustre-MDT0000 [ 1247.387621] LustreError: 35518:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1247.392814] LustreError: 35518:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 1247.612712] Lustre: server umount lustre-MDT0000 complete [ 1251.823269] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.831056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1251.833253] Lustre: Skipped 10 previous similar messages [ 1268.402803] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1268.404689] LDISKFS-fs (dm-0): recovery complete [ 1268.421745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.413793] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc3f95abb2721a5e9 [ 1277.704882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.716080] Lustre: Skipped 4 previous similar messages [ 1277.748241] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1277.762654] Lustre: Skipped 1 previous similar message [ 1279.084269] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1279.089821] Lustre: Skipped 2 previous similar messages [ 1281.546915] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1283.104818] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1283.110692] Lustre: Skipped 2 previous similar messages [ 1283.154208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1283.158346] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1287.549471] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1288.607615] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1294.667960] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 06:09:12 (1769080152) [ 1300.831159] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1302.668652] Lustre: Failing over lustre-MDT0000 [ 1302.858199] Lustre: server umount lustre-MDT0000 complete [ 1303.519845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1319.905864] Lustre: 3664:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769080161/real 1769080161] req@ffff97b27bffbb80 x1855013762187008/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769080177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1319.927857] Lustre: 3664:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1319.933161] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1319.944429] LustreError: Skipped 2 previous similar messages [ 1323.662055] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1323.664399] LDISKFS-fs (dm-0): recovery complete [ 1323.679070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.585510] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1336.018203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1336.022482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1341.774765] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1343.251989] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1351.548971] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 06:10:08 (1769080208) [ 1358.702304] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1360.618613] Lustre: Failing over lustre-MDT0000 [ 1360.885764] Lustre: server umount lustre-MDT0000 complete [ 1381.948621] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1381.951344] LDISKFS-fs (dm-0): recovery complete [ 1381.959067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.977049] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b27b805c00 x1855013762229376/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1391.289678] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1392.623039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1392.628623] Lustre: Skipped 22 previous similar messages [ 1392.900184] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1392.900465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1398.382832] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1399.703673] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1407.760381] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 06:11:04 (1769080264) [ 1414.865181] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1423.191239] Lustre: Failing over lustre-MDT0000 [ 1423.340693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1423.344261] Lustre: Skipped 1 previous similar message [ 1423.459845] Lustre: server umount lustre-MDT0000 complete [ 1442.560964] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1442.564016] LDISKFS-fs (dm-0): recovery complete [ 1442.571853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.927422] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1442.932909] Lustre: Skipped 2 previous similar messages [ 1446.216436] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1450.701474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1450.701752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1453.462387] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1454.940195] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1474.996600] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 06:12:12 (1769080332) [ 1481.375405] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1482.903233] Lustre: Failing over lustre-MDT0000 [ 1483.135854] Lustre: server umount lustre-MDT0000 complete [ 1503.361812] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1503.365021] LDISKFS-fs (dm-0): recovery complete [ 1503.370579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.859524] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b26f575f80 x1855013762361344/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1513.986655] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1515.590028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1515.590147] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1520.022328] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1521.373218] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1530.091759] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 06:13:07 (1769080387) [ 1536.082737] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1537.457117] Lustre: Failing over lustre-MDT0000 [ 1537.568231] LustreError: 44686:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1537.577787] LustreError: 44686:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 1537.681346] Lustre: server umount lustre-MDT0000 complete [ 1541.096516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.103067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1541.113608] Lustre: Skipped 20 previous similar messages [ 1556.140676] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1556.143220] LDISKFS-fs (dm-0): recovery complete [ 1556.152027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1559.668750] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1560.177175] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1560.187397] Lustre: Skipped 4 previous similar messages [ 1561.634644] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1561.638836] Lustre: Skipped 4 previous similar messages [ 1561.668052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1561.668503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1565.775979] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1567.091185] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1574.429328] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 06:13:51 (1769080431) [ 1581.632742] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1583.460635] Lustre: Failing over lustre-MDT0000 [ 1583.730090] Lustre: server umount lustre-MDT0000 complete [ 1603.554469] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769080445/real 1769080445] req@ffff97b270b3b100 x1855013762443904/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769080461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1603.576323] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1603.583076] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1603.590791] LustreError: Skipped 4 previous similar messages [ 1603.706288] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1603.711942] LDISKFS-fs (dm-0): recovery complete [ 1603.719281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.793039] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b2748d2680 x1855013762452992/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1617.727612] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1619.533420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1619.533744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1624.668829] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1625.745584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1631.685086] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 06:14:49 (1769080489) [ 1637.686213] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1639.454653] Lustre: Failing over lustre-MDT0000 [ 1639.646947] Lustre: server umount lustre-MDT0000 complete [ 1659.860597] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1659.865376] LDISKFS-fs (dm-0): recovery complete [ 1659.877061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.910821] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1665.676527] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1665.680389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1670.363558] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1671.746835] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1678.548630] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 06:15:36 (1769080536) [ 1684.029209] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1685.416662] Lustre: Failing over lustre-MDT0000 [ 1685.600857] Lustre: server umount lustre-MDT0000 complete [ 1685.993814] LustreError: 10309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.015652] LustreError: 10309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 257 previous similar messages [ 1704.867710] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1704.869606] LDISKFS-fs (dm-0): recovery complete [ 1704.877958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.122187] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1705.127170] Lustre: Skipped 4 previous similar messages [ 1707.620365] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1710.648901] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1710.649757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1713.825531] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1715.150183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1721.785690] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 06:16:19 (1769080579) [ 1727.312666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1728.767217] Lustre: Failing over lustre-MDT0000 [ 1728.944376] Lustre: server umount lustre-MDT0000 complete [ 1731.041921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1747.090605] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1747.092596] LDISKFS-fs (dm-0): recovery complete [ 1747.099652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1749.930988] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1752.613643] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1752.619254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1755.074372] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1756.204316] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1762.979622] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 06:17:00 (1769080620) [ 1768.509832] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1770.012396] Lustre: Failing over lustre-MDT0000 [ 1770.256604] Lustre: server umount lustre-MDT0000 complete [ 1789.243717] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1789.246393] LDISKFS-fs (dm-0): recovery complete [ 1789.252832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.514669] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1794.659545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1794.667801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1797.759541] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1798.765826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1804.729264] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 06:17:42 (1769080662) [ 1810.363424] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1811.661839] Lustre: Failing over lustre-MDT0000 [ 1811.841544] Lustre: server umount lustre-MDT0000 complete [ 1831.141802] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1831.144547] LDISKFS-fs (dm-0): recovery complete [ 1831.149729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.407814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1831.414130] Lustre: Skipped 10 previous similar messages [ 1834.254774] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1836.589114] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1836.589973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1839.862345] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1840.999313] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1846.554322] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 06:18:24 (1769080704) [ 1851.607214] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1852.790915] Lustre: Failing over lustre-MDT0000 [ 1852.995835] Lustre: server umount lustre-MDT0000 complete [ 1870.193451] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1870.194872] LDISKFS-fs (dm-0): recovery complete [ 1870.198571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.171768] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1875.523052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1875.523292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1879.195734] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1880.213454] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1885.755978] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 06:19:03 (1769080743) [ 1890.823095] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1892.359813] Lustre: Failing over lustre-MDT0000 [ 1892.497862] Lustre: server umount lustre-MDT0000 complete [ 1910.885689] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1910.888350] LDISKFS-fs (dm-0): recovery complete [ 1910.895356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.422058] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1916.386455] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1916.391544] Lustre: Skipped 43 previous similar messages [ 1916.441775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 1916.446166] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 1920.194186] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1921.454796] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1928.074776] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 06:19:45 (1769080785) [ 1932.837984] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1934.831307] Lustre: Failing over lustre-MDT0000 [ 1934.971608] Lustre: server umount lustre-MDT0000 complete [ 1952.978895] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1952.981270] LDISKFS-fs (dm-0): recovery complete [ 1952.992408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.463806] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b27b599880 x1855013762677120/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1965.017638] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1968.177303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1968.186884] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1970.435204] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1971.633683] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1976.919370] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 06:20:34 (1769080834) [ 1981.342607] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1982.673323] Lustre: Failing over lustre-MDT0000 [ 1982.787838] LustreError: 62940:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff97b26f357f00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1982.870903] Lustre: server umount lustre-MDT0000 complete [ 2000.815215] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2000.816761] LDISKFS-fs (dm-0): recovery complete [ 2000.826503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.056230] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b274a22680 x1855013762706304/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2011.437975] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2014.763949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 2014.763966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 2017.122987] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2018.367499] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2024.703309] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 06:21:22 (1769080882) [ 2029.955189] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2031.051140] Lustre: Failing over lustre-MDT0000 [ 2031.193514] Lustre: server umount lustre-MDT0000 complete [ 2035.171249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2035.178985] LustreError: Skipped 5 previous similar messages [ 2048.709622] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2048.711162] LDISKFS-fs (dm-0): recovery complete [ 2048.720809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.063975] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2054.194334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 2054.199395] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:993) [ 2056.314164] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2057.278298] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2062.567210] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 06:22:00 (1769080920) [ 2066.665763] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2067.996517] Lustre: Failing over lustre-MDT0000 [ 2068.098156] LustreError: 66584:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2068.100351] LustreError: 66584:0:(obd_class.h:479:obd_check_dev()) Skipped 95 previous similar messages [ 2068.132812] Lustre: server umount lustre-MDT0000 complete [ 2069.472442] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2069.481478] Lustre: Skipped 47 previous similar messages [ 2086.255037] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2086.257314] LDISKFS-fs (dm-0): recovery complete [ 2086.264613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.590025] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b27c173b80 x1855013762762240/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2096.746704] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2096.750828] Lustre: Skipped 11 previous similar messages [ 2097.882514] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2101.283549] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2101.289479] Lustre: Skipped 11 previous similar messages [ 2101.309861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2101.311224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2103.618607] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2104.645918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2110.407709] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 06:22:47 (1769080967) [ 2115.473772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2117.038897] Lustre: Failing over lustre-MDT0000 [ 2117.256852] Lustre: server umount lustre-MDT0000 complete [ 2134.433394] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2134.435392] LDISKFS-fs (dm-0): recovery complete [ 2134.440454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.511525] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2134.515947] LustreError: Skipped 11 previous similar messages [ 2137.362396] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2140.200848] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2140.201437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2142.504303] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2143.665359] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2150.034868] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 06:23:27 (1769081007) [ 2155.066991] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2156.270945] Lustre: Failing over lustre-MDT0000 [ 2156.449274] Lustre: server umount lustre-MDT0000 complete [ 2173.937491] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2173.938886] LDISKFS-fs (dm-0): recovery complete [ 2173.944338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.542957] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2179.616841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2179.617284] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2181.509652] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2182.458203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2187.701348] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 06:24:05 (1769081045) [ 2190.190259] Lustre: 71714:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c24165d9-4026-4158-9006-fa0818e10e68 at adminstrative request [ 2193.888654] Lustre: Failing over lustre-MDT0000 [ 2194.117179] Lustre: server umount lustre-MDT0000 complete [ 2210.287717] Lustre: 3666:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769081052/real 1769081052] req@ffff97b270108000 x1855013762836864/t0(0) o400->MGC192.168.204.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1769081068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2210.342215] Lustre: 3666:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2212.826574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.899760] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2220.904764] Lustre: Skipped 11 previous similar messages [ 2226.260479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2226.261198] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2226.871643] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2235.448509] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2236.546474] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2240.579937] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2246.413212] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2252.227834] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 06:25:09 (1769081109) [ 2253.118165] Lustre: 73563:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c24165d9-4026-4158-9006-fa0818e10e68 at adminstrative request [ 2262.959835] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 06:25:20 (1769081120) [ 2269.721636] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2271.412358] Lustre: Failing over lustre-MDT0000 [ 2271.685391] Lustre: server umount lustre-MDT0000 complete [ 2287.594713] LustreError: 6534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.614533] LustreError: 6534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 300 previous similar messages [ 2293.412599] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2293.414523] LDISKFS-fs (dm-0): recovery complete [ 2293.421697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.848796] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b27bc7aa00 x1855013762887680/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2298.880726] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [ 2302.810580] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2304.655824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 2304.655862] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 2309.362747] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2310.850200] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2318.088628] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 06:26:15 (1769081175) [ 2324.701769] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2326.390240] Lustre: Failing over lustre-MDT0000 [ 2326.588293] Lustre: server umount lustre-MDT0000 complete [ 2347.737762] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2347.740145] LDISKFS-fs (dm-0): recovery complete [ 2347.750336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.953633] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2362.493214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2362.498628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2367.281781] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2368.696088] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2376.877325] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 06:27:13 (1769081233) [ 2382.889793] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2384.286815] Lustre: Failing over lustre-MDT0000 [ 2384.456474] Lustre: server umount lustre-MDT0000 complete [ 2404.340179] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2404.344271] LDISKFS-fs (dm-0): recovery complete [ 2404.355527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.678847] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (not set up) [ 2413.686120] Lustre: Skipped 2 previous similar messages [ 2417.736897] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2419.293703] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2419.298406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2424.461733] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2425.821197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2433.436686] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 06:28:10 (1769081290) [ 2439.992218] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2441.993263] Lustre: Failing over lustre-MDT0000 [ 2442.176222] Lustre: server umount lustre-MDT0000 complete [ 2463.889803] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2463.892668] LDISKFS-fs (dm-0): recovery complete [ 2463.903629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.107390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.111271] Lustre: Skipped 12 previous similar messages [ 2474.526696] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2476.595619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2476.596095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2480.539336] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2481.800703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2488.181175] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 06:29:05 (1769081345) [ 2494.321368] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2495.803421] Lustre: Failing over lustre-MDT0000 [ 2496.032358] Lustre: server umount lustre-MDT0000 complete [ 2519.258608] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2519.263817] LDISKFS-fs (dm-0): recovery complete [ 2519.278673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.203698] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2529.253515] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2529.259601] Lustre: Skipped 47 previous similar messages [ 2529.334585] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 2529.334756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 2535.328960] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2536.777676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2544.410418] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 06:30:01 (1769081401) [ 2550.145989] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2551.944909] Lustre: Failing over lustre-MDT0000 [ 2552.248436] Lustre: server umount lustre-MDT0000 complete [ 2554.852627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2554.863349] LustreError: Skipped 7 previous similar messages [ 2572.844272] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2572.849126] LDISKFS-fs (dm-0): recovery complete [ 2572.855497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.570455] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (not set up) [ 2587.331433] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2588.330540] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2588.330540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2594.158321] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2595.800255] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2602.607269] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 06:30:59 (1769081459) [ 2609.446403] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2611.160326] Lustre: Failing over lustre-MDT0000 [ 2611.394468] Lustre: server umount lustre-MDT0000 complete [ 2630.058588] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2630.061416] LDISKFS-fs (dm-0): recovery complete [ 2630.070597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.840390] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b270590e00 x1855013763086464/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2639.862980] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 2639.981072] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (not set up) [ 2643.363100] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2645.575948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 2645.580967] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 2649.575764] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2650.852052] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2658.096485] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 06:31:55 (1769081515) [ 2665.787393] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2667.761491] Lustre: Failing over lustre-MDT0000 [ 2668.054870] Lustre: server umount lustre-MDT0000 complete [ 2671.073800] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.083747] Lustre: Skipped 43 previous similar messages [ 2688.608336] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2688.616796] LDISKFS-fs (dm-0): recovery complete [ 2688.635192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2698.292555] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2698.301886] Lustre: Skipped 10 previous similar messages [ 2701.134429] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2703.383935] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2703.392286] Lustre: Skipped 10 previous similar messages [ 2703.417096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 2703.418058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 2706.762783] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2707.955138] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2714.597393] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 06:32:51 (1769081571) [ 2720.270456] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2721.535541] Lustre: Failing over lustre-MDT0000 [ 2721.628128] LustreError: 88870:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2721.631659] LustreError: 88870:0:(obd_class.h:479:obd_check_dev()) Skipped 95 previous similar messages [ 2721.687822] Lustre: server umount lustre-MDT0000 complete [ 2739.167183] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2739.172885] LustreError: Skipped 10 previous similar messages [ 2739.284629] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2739.289200] LDISKFS-fs (dm-0): recovery complete [ 2739.295916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.705639] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2755.122041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 2755.123419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 2757.506025] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2758.577172] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2764.070745] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 06:33:41 (1769081621) [ 2768.390207] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2769.577521] Lustre: Failing over lustre-MDT0000 [ 2769.782651] Lustre: server umount lustre-MDT0000 complete [ 2786.918342] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2786.920485] LDISKFS-fs (dm-0): recovery complete [ 2786.937960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2798.817993] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2801.677436] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 2801.681220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 2803.938750] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2804.980229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2810.570742] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 06:34:28 (1769081668) [ 2815.284593] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2816.437857] Lustre: Failing over lustre-MDT0000 [ 2816.625247] Lustre: server umount lustre-MDT0000 complete [ 2833.631791] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2833.633974] LDISKFS-fs (dm-0): recovery complete [ 2833.640511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.844795] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2833.857733] Lustre: Skipped 10 previous similar messages [ 2836.006421] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2839.059123] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 2839.059359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 2840.791388] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2841.565810] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2845.872761] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 06:35:03 (1769081703) [ 2846.324417] Lustre: 93896:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c24165d9-4026-4158-9006-fa0818e10e68 at adminstrative request [ 2851.873496] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 06:35:09 (1769081709) [ 2855.200386] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2856.074984] Lustre: Failing over lustre-MDT0000 [ 2856.220413] Lustre: server umount lustre-MDT0000 complete [ 2862.136321] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2862.138485] LDISKFS-fs (dm-0): recovery complete [ 2862.143541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.386397] Lustre: lustre-MDT0000: Aborting client recovery [ 2862.388548] LustreError: 95175:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2862.393733] Lustre: 95208:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2862.399118] Lustre: 95208:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c24165d9-4026-4158-9006-fa0818e10e68@ [ 2862.405219] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2862.409455] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2862.414375] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2862.447234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 2862.447366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 2864.208497] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2867.687448] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2878.514427] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 06:35:36 (1769081736) [ 2882.334083] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2883.252156] Lustre: Failing over lustre-MDT0000 [ 2883.380581] Lustre: server umount lustre-MDT0000 complete [ 2888.161583] LustreError: 42331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.170132] LustreError: 42331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 334 previous similar messages [ 2889.711412] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2889.713029] LDISKFS-fs (dm-0): recovery complete [ 2889.719972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.905362] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2889.937628] Lustre: lustre-MDT0000: Aborting client recovery [ 2889.941535] LustreError: 97021:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2889.953064] Lustre: 97054:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2889.958204] Lustre: 97054:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2889.963279] Lustre: 97054:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2889.968906] Lustre: 97054:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2889.973855] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2889.979673] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 2889.989701] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 2890.018355] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1537) [ 2890.019252] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1537) [ 2892.074492] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2895.331700] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2903.727745] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2903.730484] Lustre: Skipped 1 previous similar message [ 2907.926371] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 06:36:05 (1769081765) [ 2912.277284] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2913.270451] Lustre: Failing over lustre-MDT0000 [ 2913.450717] Lustre: server umount lustre-MDT0000 complete [ 2920.902438] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2920.904429] LDISKFS-fs (dm-0): recovery complete [ 2920.911363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2921.132282] Lustre: lustre-MDT0000: Aborting client recovery [ 2921.134490] LustreError: 98869:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2921.138814] Lustre: 98902:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2921.144458] Lustre: 98902:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2921.147603] Lustre: 98902:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2921.153012] Lustre: 98902:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2921.156284] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2921.161324] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 2921.175221] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 2921.201830] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 2921.201837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 2923.291395] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2926.563729] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2936.843297] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 06:36:34 (1769081794) [ 2937.327526] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2937.329467] LustreError: 8443:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b274ae8000 x1855013744919680/t201863462916(0) o36->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:536/0 lens 512/456 e 0 to 0 dl 1769081806 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 2940.269843] Lustre: Failing over lustre-MDT0000 [ 2940.404561] Lustre: server umount lustre-MDT0000 complete [ 2944.764816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.980938] Lustre: lustre-MDT0000: Aborting client recovery [ 2944.982572] LustreError: 100176:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2944.986076] Lustre: 100209:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2944.990086] Lustre: 100209:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2944.993966] Lustre: 100209:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2944.997536] Lustre: 100209:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2945.001285] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2945.007174] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 2945.016372] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 2945.053702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1601) [ 2945.059262] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 2947.160359] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2950.120107] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2960.957262] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 2961.871989] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 06:36:59 (1769081819) [ 2966.009912] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2967.543679] Lustre: Failing over lustre-MDT0000 [ 2967.680281] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (stopping) [ 2967.765931] Lustre: server umount lustre-MDT0000 complete [ 2974.125387] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2974.127954] LDISKFS-fs (dm-0): recovery complete [ 2974.133074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.369245] Lustre: lustre-MDT0000: Aborting client recovery [ 2974.373159] LustreError: 102122:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2974.377808] Lustre: 102156:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2974.384989] Lustre: 102156:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2974.389400] Lustre: 102156:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2974.394570] Lustre: 102156:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2974.400908] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2974.405791] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 2974.418687] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 2974.467066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1633) [ 2974.468946] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 2976.464484] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2979.814103] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2990.714350] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 06:37:28 (1769081848) [ 3010.751102] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3011.859933] Lustre: Failing over lustre-MDT0000 [ 3011.888563] LustreError: 3664:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff97b274a20380 x1855013763400576/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3012.021242] Lustre: server umount lustre-MDT0000 complete [ 3027.940885] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3027.944579] LDISKFS-fs (dm-0): recovery complete [ 3027.955398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3030.219263] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3033.645652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3033.645728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3035.310403] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3036.056416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3047.589784] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 06:38:25 (1769081905) [ 3059.369754] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3064.807567] Lustre: Failing over lustre-MDT0000 [ 3064.860258] LustreError: 3665:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff97b27c3f5180 x1855013763525760/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3064.869094] LustreError: 3665:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 3064.946379] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (stopping) [ 3065.073803] Lustre: server umount lustre-MDT0000 complete [ 3080.215056] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3080.216217] LDISKFS-fs (dm-0): recovery complete [ 3080.219496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.302307] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (not set up) [ 3080.367448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.370838] Lustre: Skipped 13 previous similar messages [ 3082.293734] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3087.137726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3087.137850] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3088.733219] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3089.554175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3101.406729] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 06:39:19 (1769081959) [ 3102.318411] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3102.777892] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3102.782358] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3105.749919] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 06:39:23 (1769081963) [ 3118.171631] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3123.422255] Lustre: Failing over lustre-OST0000 [ 3123.480272] Lustre: server umount lustre-OST0000 complete [ 3138.912762] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3138.915309] LDISKFS-fs (dm-2): recovery complete [ 3138.921405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3141.146242] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3141.150745] Lustre: Skipped 56 previous similar messages [ 3141.568597] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3189.368554] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 06:40:47 (1769082047) [ 3192.577916] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3193.774047] Lustre: Failing over lustre-MDT0000 [ 3193.923714] Lustre: server umount lustre-MDT0000 complete [ 3194.847638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3194.850169] LustreError: Skipped 20 previous similar messages [ 3208.953414] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3208.955626] LDISKFS-fs (dm-0): recovery complete [ 3208.959547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.911130] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3214.346965] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3214.347000] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3215.875936] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3216.638167] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3230.687396] Lustre: 111227:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769082072/real 1769082072] req@ffff97b27c3a6680 x1855013763882624/t0(0) o5->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1769082088 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-0-0.0' uid:0 gid:0 projid:4294967295 [ 3230.698440] Lustre: 111227:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 3230.702606] LustreError: 111227:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3230.703307] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3231.177509] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 06:41:29 (1769082089) [ 3231.712427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3233.384770] LustreError: 10309:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3238.879181] LustreError: 10309:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3238.882483] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnecting [ 3239.741947] LustreError: 10309:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3245.023144] LustreError: 10309:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3245.027343] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnecting [ 3245.785854] LustreError: 6534:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3251.167134] LustreError: 6534:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3251.175715] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnecting [ 3252.035147] LustreError: 6534:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3257.312043] LustreError: 6534:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3258.082524] LustreError: 81679:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3263.455128] LustreError: 81679:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3263.458686] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnecting [ 3263.461817] Lustre: Skipped 1 previous similar message [ 3270.390880] LustreError: 81679:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3270.394142] LustreError: 81679:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3275.743118] LustreError: 81679:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3275.746805] LustreError: 81679:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3281.888226] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnecting [ 3281.892659] Lustre: Skipped 2 previous similar messages [ 3288.756872] LustreError: 81679:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3288.760226] LustreError: 81679:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3294.175158] LustreError: 81679:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3294.178231] LustreError: 81679:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3297.400090] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 06:42:35 (1769082155) [ 3298.117369] LustreError: 6534:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3307.625114] Lustre: lustre-MDT0000: Export ffff97b245cbc000 already connecting from 192.168.204.51@tcp [ 3312.745392] Lustre: lustre-MDT0000: Export ffff97b245cbc000 already connecting from 192.168.204.51@tcp [ 3317.868482] Lustre: lustre-MDT0000: Export ffff97b245cbc000 already connecting from 192.168.204.51@tcp [ 3322.985579] Lustre: lustre-MDT0000: Export ffff97b245cbc000 already connecting from 192.168.204.51@tcp [ 3322.989588] Lustre: Skipped 1 previous similar message [ 3328.105247] Lustre: lustre-MDT0000: Export ffff97b245cbc000 already connecting from 192.168.204.51@tcp [ 3338.159140] LustreError: 6534:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3338.163925] Lustre: 6534:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff97b2748da680 x1855013747556736/t0(0) o38->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:0/0 lens 520/416 e 0 to 0 dl 1769082176 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3338.346763] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnecting [ 3338.351504] Lustre: Skipped 3 previous similar messages [ 3338.353428] LustreError: 8443:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3362.921412] Lustre: lustre-MDT0000: Export ffff97b245cbc000 already connecting from 192.168.204.51@tcp [ 3362.924666] Lustre: Skipped 1 previous similar message [ 3378.407121] LustreError: 8443:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3378.410614] Lustre: 8443:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff97b26f4f9500 x1855013747561984/t0(0) o38->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:0/0 lens 520/416 e 0 to 0 dl 1769082216 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3382.282890] LustreError: 8443:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3406.954637] Lustre: lustre-MDT0000: Export ffff97b245cbc000 already connecting from 192.168.204.51@tcp [ 3406.960869] Lustre: Skipped 3 previous similar messages [ 3422.343146] LustreError: 8443:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3422.346213] Lustre: 8443:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff97b2701c0380 x1855013747565568/t0(0) o38->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:0/0 lens 520/416 e 0 to 0 dl 1769082260 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3424.239243] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnecting [ 3424.243197] Lustre: Skipped 1 previous similar message [ 3424.245120] LustreError: 8443:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3448.937263] Lustre: lustre-MDT0000: Export ffff97b245cbc000 already connecting from 192.168.204.51@tcp [ 3448.940256] Lustre: Skipped 3 previous similar messages [ 3464.287126] LustreError: 8443:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3464.289909] Lustre: 8443:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff97b14491ce00 x1855013747569024/t0(0) o38->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:0/0 lens 520/416 e 0 to 0 dl 1769082302 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3464.297265] LustreError: 10309:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3504.343134] LustreError: 10309:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3504.346828] Lustre: 10309:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff97b27c3a6d80 x1855013747571968/t0(0) o38->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:0/0 lens 520/416 e 0 to 0 dl 1769082342 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3509.584026] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 06:46:07 (1769082367) [ 3512.282547] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3513.983041] Lustre: Failing over lustre-MDT0000 [ 3514.052049] LustreError: 115313:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3514.054364] LustreError: 115313:0:(obd_class.h:479:obd_check_dev()) Skipped 91 previous similar messages [ 3514.089889] Lustre: server umount lustre-MDT0000 complete [ 3514.494493] LustreError: 10309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.501816] LustreError: 10309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 75 previous similar messages [ 3516.383455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.387269] Lustre: Skipped 49 previous similar messages [ 3518.733350] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3518.735312] LDISKFS-fs (dm-0): recovery complete [ 3518.739619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3518.783908] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3518.788456] LustreError: Skipped 10 previous similar messages [ 3518.841595] Lustre: *** cfs_fail_loc=712, val=0*** [ 3518.844425] LustreError: 42168:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff97b27c00dc00 x1855013764031872/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 3518.894336] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3518.894443] Lustre: lustre-MDT0000: Aborting client recovery [ 3518.896608] Lustre: Skipped 19 previous similar messages [ 3518.899705] LustreError: 115958:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3518.903078] Lustre: 115991:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3518.905964] Lustre: 115991:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3518.909133] Lustre: 115991:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3518.912696] Lustre: 115991:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3518.915500] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3518.918733] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 3518.924172] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3518.946354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 3518.946358] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 3520.373670] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3524.065169] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3530.689213] Lustre: Failing over lustre-MDT0000 [ 3530.848821] Lustre: server umount lustre-MDT0000 complete [ 3543.844394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3545.193311] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3545.196808] Lustre: Skipped 7 previous similar messages [ 3545.261528] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3549.159842] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3549.163089] Lustre: Skipped 7 previous similar messages [ 3549.177804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 3549.177847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 3550.358527] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3550.964939] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3554.579694] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 06:46:52 (1769082412) [ 3554.615323] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnecting [ 3554.618957] Lustre: Skipped 2 previous similar messages [ 3557.858546] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 06:46:55 (1769082415) [ 3558.222536] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3558.224323] LustreError: 117718:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b27c00d180 x1855013747628416/t0(0) o700->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:402/0 lens 264/248 e 0 to 0 dl 1769082427 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 3575.806459] Lustre: Failing over lustre-MDT0000 [ 3575.908161] Lustre: server umount lustre-MDT0000 complete [ 3589.627196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3591.192337] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3595.253425] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 3595.253454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 3596.519150] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3597.142135] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3601.550715] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 06:47:39 (1769082459) [ 3602.514761] Lustre: Failing over lustre-OST0000 [ 3602.578235] Lustre: server umount lustre-OST0000 complete [ 3615.550349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3617.757305] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3620.955345] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3621.535362] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3686.787223] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 06:49:04 (1769082544) [ 3689.436803] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3690.241898] Lustre: Failing over lustre-MDT0000 [ 3690.381605] Lustre: server umount lustre-MDT0000 complete [ 3704.409213] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3704.411094] LDISKFS-fs (dm-0): recovery complete [ 3704.416038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.517804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.520523] Lustre: Skipped 6 previous similar messages [ 3705.810074] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3710.008226] Lustre: *** cfs_fail_loc=216, val=0*** [ 3710.008236] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 3710.014705] LustreError: 121769:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 3711.071456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 3772.628524] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 06:50:30 (1769082630) [ 3773.265434] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3773.269206] Lustre: Skipped 24 previous similar messages [ 3780.563082] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 06:50:38 (1769082638) [ 3781.528818] Lustre: Failing over lustre-MDT0000 [ 3781.600392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3781.602554] Lustre: Skipped 3 previous similar messages [ 3781.752728] Lustre: server umount lustre-MDT0000 complete [ 3794.625568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.053524] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3800.036315] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 3800.037670] LustreError: 123331:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b2748d1f80 x1855013747748096/t0(0) o101->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:683/0 lens 328/344 e 0 to 0 dl 1769082708 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 3855.477536] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [ 3855.482624] Lustre: 123331:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3855.501741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 3855.501753] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 3856.720340] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3857.306779] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3860.867185] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 06:51:58 (1769082718) [ 3862.208473] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3864.922391] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3865.513376] Lustre: Failing over lustre-MDT0000 [ 3865.668801] Lustre: server umount lustre-MDT0000 complete [ 3866.079448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3866.081950] LustreError: Skipped 6 previous similar messages [ 3879.431732] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3879.433770] LDISKFS-fs (dm-0): recovery complete [ 3879.436537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3880.749061] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3885.045432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3137) [ 3885.045501] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 3886.111822] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3886.593190] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3889.829859] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 06:52:27 (1769082747) [ 3890.184672] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3893.677395] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3894.274979] Lustre: Failing over lustre-MDT0000 [ 3894.365626] Lustre: server umount lustre-MDT0000 complete [ 3908.376541] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3908.377573] LDISKFS-fs (dm-0): recovery complete [ 3908.379928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3909.712298] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3913.713762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 3913.713867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3137) [ 3914.780223] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3915.253061] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3918.509467] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 06:52:56 (1769082776) [ 3918.886716] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3922.455482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3923.018726] Lustre: Failing over lustre-MDT0000 [ 3923.103328] Lustre: server umount lustre-MDT0000 complete [ 3937.254452] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3937.255623] LDISKFS-fs (dm-0): recovery complete [ 3937.258666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.617952] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3942.390169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3169) [ 3942.390209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 3946.920257] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 06:53:24 (1769082804) [ 3948.320275] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3948.322608] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 3948.325049] LustreError: 6536:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b14491d880 x1855013747799424/t257698037777(0) o35->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:37/0 lens 392/456 e 0 to 0 dl 1769082817 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3949.249924] Lustre: Failing over lustre-MDT0000 [ 3949.408524] Lustre: server umount lustre-MDT0000 complete [ 3962.331510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.843400] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3967.471269] Lustre: 6536:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b274a21500 x1855013747799424/t257698037777(0) o35->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:56/0 lens 392/456 e 0 to 0 dl 1769082836 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3967.479045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3233) [ 3967.479077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 3968.674979] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3969.241100] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3972.611250] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 06:53:50 (1769082830) [ 3972.976788] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3972.978323] LustreError: 8443:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b27b557480 x1855013747812608/t261993005072(0) o36->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:106/0 lens 504/448 e 0 to 0 dl 1769082886 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3976.751546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3977.373455] Lustre: Failing over lustre-MDT0000 [ 3977.461485] Lustre: server umount lustre-MDT0000 complete [ 3991.577966] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3991.579705] LDISKFS-fs (dm-0): recovery complete [ 3991.583678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3993.029579] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3997.167931] Lustre: 10309:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b27032f800 x1855013747812608/t261993005072(0) o36->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:130/0 lens 504/2880 e 0 to 0 dl 1769082910 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3997.174875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 3997.174939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3265) [ 3998.358990] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3998.882902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4002.276242] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 06:54:20 (1769082860) [ 4002.630542] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4002.632040] LustreError: 8443:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b27c304e00 x1855013747825536/t266287972368(0) o36->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:135/0 lens 504/448 e 0 to 0 dl 1769082915 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4003.905750] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4006.353975] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4006.993058] Lustre: Failing over lustre-MDT0000 [ 4007.085542] Lustre: server umount lustre-MDT0000 complete [ 4021.173608] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4021.175072] LDISKFS-fs (dm-0): recovery complete [ 4021.177767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4022.570412] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4026.350771] Lustre: 10309:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b27c04b480 x1855013747825536/t266287972368(0) o36->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:159/0 lens 504/2880 e 0 to 0 dl 1769082939 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4026.357600] Lustre: 10309:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4026.358456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 4026.358466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3297) [ 4030.722403] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 06:54:48 (1769082888) [ 4031.085241] LustreError: 6532:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b27079c700 x1855013747838208/t270582939664(0) o36->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:164/0 lens 504/448 e 0 to 0 dl 1769082944 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4031.093876] LustreError: 6532:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 4032.362806] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4032.365459] Lustre: Skipped 1 previous similar message [ 4035.141563] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4035.790406] Lustre: Failing over lustre-MDT0000 [ 4035.885771] Lustre: server umount lustre-MDT0000 complete [ 4049.993211] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4049.994455] LDISKFS-fs (dm-0): recovery complete [ 4049.996968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.430857] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4055.534305] Lustre: 6532:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b27c187480 x1855013747838208/t270582939664(0) o36->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:188/0 lens 504/2880 e 0 to 0 dl 1769082968 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4055.539951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3329) [ 4055.540048] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 4059.648893] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 06:55:17 (1769082917) [ 4060.012915] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4061.300149] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4061.301411] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4061.303448] Lustre: Skipped 2 previous similar messages [ 4061.305073] LustreError: 6536:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b27bc8d500 x1855013747850112/t274877906960(0) o35->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:150/0 lens 392/456 e 0 to 0 dl 1769082930 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4064.805472] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4065.447776] Lustre: Failing over lustre-MDT0000 [ 4065.540639] Lustre: server umount lustre-MDT0000 complete [ 4079.796543] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4079.798145] LDISKFS-fs (dm-0): recovery complete [ 4079.802272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.172870] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4085.230996] Lustre: 6536:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b27032dc00 x1855013747850112/t274877906960(0) o35->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:174/0 lens 392/456 e 0 to 0 dl 1769082954 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4085.240865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3361) [ 4085.241124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 4089.616120] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 06:55:47 (1769082947) [ 4089.940862] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4150.386771] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnecting [ 4150.394206] Lustre: Skipped 4 previous similar messages [ 4150.406731] Lustre: 6532:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b274883480 x1855013747860352/t279172874255(0) o101->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:283/0 lens 664/3488 e 0 to 0 dl 1769083063 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4153.636158] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 06:56:51 (1769083011) [ 4156.717616] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4157.502789] Lustre: Failing over lustre-MDT0000 [ 4157.573931] LustreError: 139265:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4157.575933] LustreError: 139265:0:(obd_class.h:479:obd_check_dev()) Skipped 107 previous similar messages [ 4157.598480] Lustre: server umount lustre-MDT0000 complete [ 4160.622987] LustreError: 81679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4160.632566] LustreError: 81679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 171 previous similar messages [ 4162.015707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.027563] Lustre: Skipped 58 previous similar messages [ 4171.666478] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4171.667926] LDISKFS-fs (dm-0): recovery complete [ 4171.672640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4171.706915] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4171.710551] LustreError: Skipped 12 previous similar messages [ 4171.788048] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4171.790266] Lustre: Skipped 15 previous similar messages [ 4173.128761] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4175.977929] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4175.981388] Lustre: Skipped 12 previous similar messages [ 4176.876428] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4176.879438] Lustre: Skipped 12 previous similar messages [ 4176.894561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 4176.894580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3393) [ 4178.054527] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4178.657320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4192.329707] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 06:57:30 (1769083050) [ 4195.620569] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4196.328235] Lustre: Failing over lustre-MDT0000 [ 4196.458469] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (stopping) [ 4196.490222] Lustre: server umount lustre-MDT0000 complete [ 4210.835347] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4210.837053] LDISKFS-fs (dm-0): recovery complete [ 4210.839991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4212.215562] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4216.318751] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 4216.318842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3425) [ 4217.559195] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4218.105424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4220.392695] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4225.502696] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 06:58:03 (1769083083) [ 4235.358802] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4235.974104] Lustre: Failing over lustre-MDT0000 [ 4236.224319] Lustre: server umount lustre-MDT0000 complete [ 4250.056076] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4250.057225] LDISKFS-fs (dm-0): recovery complete [ 4250.060166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.441348] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4255.277075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 4255.277095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 4256.290895] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4256.779105] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4269.651791] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 06:58:47 (1769083127) [ 4273.207066] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4273.837030] Lustre: Failing over lustre-MDT0000 [ 4273.957304] Lustre: server umount lustre-MDT0000 complete [ 4287.805143] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4287.806598] LDISKFS-fs (dm-0): recovery complete [ 4287.809167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.103098] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4293.117417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 4293.117425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 4294.294887] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4294.837670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4297.715663] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 4298.303724] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 4300.449864] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 06:59:18 (1769083158) [ 4306.278130] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 4367.332887] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4367.334089] Lustre: Skipped 1 previous similar message [ 4367.334897] LustreError: 6533:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b271393450 x1855013750648704/t296352743435(0) o36->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:500/0 lens 66040/440 e 0 to 0 dl 1769083280 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4367.339029] LustreError: 6533:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 4426.866710] Lustre: 10309:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b2705cd500 x1855013750648704/t296352743435(0) o36->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:559/0 lens 66040/440 e 0 to 0 dl 1769083339 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4430.499547] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 4431.131712] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 07:01:28 (1769083288) [ 4436.333503] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4437.185890] Lustre: Failing over lustre-MDT0000 [ 4437.353427] Lustre: server umount lustre-MDT0000 complete [ 4452.514990] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4452.517162] LDISKFS-fs (dm-0): recovery complete [ 4452.523699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.681673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4452.684033] Lustre: Skipped 13 previous similar messages [ 4454.247858] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4457.953951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4457.959122] Lustre: Skipped 53 previous similar messages [ 4458.409898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 4458.409909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 4459.848639] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4460.541531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4465.652861] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 07:02:03 (1769083323) [ 4473.300956] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4477.633224] Lustre: Failing over lustre-OST0000 [ 4477.680708] Lustre: server umount lustre-OST0000 complete [ 4492.573966] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 4492.575864] LDISKFS-fs (dm-2): recovery complete [ 4492.584607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4494.865349] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4506.847226] Lustre: Failing over lustre-OST0000 [ 4506.909180] Lustre: server umount lustre-OST0000 complete [ 4520.423205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4522.615966] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4525.963505] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4526.545991] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4560.754729] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 07:03:38 (1769083418) [ 4561.711488] Lustre: Failing over lustre-MDT0000 [ 4561.966955] Lustre: server umount lustre-MDT0000 complete [ 4563.938089] LustreError: 6517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769083421 with bad export cookie 14121417866459473004 [ 4574.889280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4576.379276] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4580.339603] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 4580.339603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 4587.865105] Lustre: Failing over lustre-MDT0000 [ 4588.076140] Lustre: server umount lustre-MDT0000 complete [ 4590.559403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4590.561721] LustreError: Skipped 12 previous similar messages [ 4600.998741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4602.342705] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4606.452590] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 4606.452590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 4607.735947] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4608.332186] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4611.884486] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 07:04:29 (1769083469) [ 4623.079851] Lustre: Failing over lustre-OST0000 [ 4623.118970] Lustre: server umount lustre-OST0000 complete [ 4635.843145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4637.752633] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4640.753327] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4641.264603] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4644.870635] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 07:05:02 (1769083502) [ 4645.491869] Lustre: Failing over lustre-MDT0000 [ 4645.801111] Lustre: server umount lustre-MDT0000 complete [ 4648.763792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4648.793572] Lustre: *** cfs_fail_loc=605, val=0*** [ 4648.794720] LustreError: 157037:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0a3bab0 failed: rc = -95 [ 4648.797721] LustreError: 157037:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 4648.799882] LustreError: 157037:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 4648.802089] LustreError: 157037:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 4648.803938] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 4648.805748] LustreError: 157037:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 4648.809530] Lustre: server umount lustre-MDT0000 complete [ 4648.810857] LustreError: 157037:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 4651.182527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4652.489900] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4655.023466] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 07:05:12 (1769083512) [ 4656.626185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 4656.626205] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 4658.973252] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4659.918889] Lustre: Failing over lustre-MDT0000 [ 4660.145933] Lustre: server umount lustre-MDT0000 complete [ 4674.147940] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4674.149480] LDISKFS-fs (dm-0): recovery complete [ 4674.153360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.563752] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4677.741635] Lustre: *** cfs_fail_loc=707, val=0*** [ 4738.170738] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 4738.204695] Lustre: 159191:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4738.213612] Lustre: 159191:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 4738.461559] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 4738.461721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 4741.708888] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4742.882363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4750.341737] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 07:06:47 (1769083607) [ 4782.109613] LustreError: 6533:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff97b14a7fb800 x1855013751530496/t0(0) o101->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:116/0 lens 664/0 e 0 to 0 dl 1769083651 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4782.143129] LustreError: 6533:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 4793.207124] LustreError: 6533:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4793.232352] LustreError: 6536:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff97b14a7fbb80 x1855013751531776/t0(0) o35->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:127/0 lens 392/0 e 0 to 0 dl 1769083662 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4794.702987] LustreError: 8443:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff97b27bcba680 x1855013751537664/t0(0) o101->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:167/0 lens 576/0 e 0 to 0 dl 1769083702 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 4794.722594] LustreError: 8443:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 4798.954125] LustreError: 16652:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff97b274a21c00 x1855013765690240/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:133/0 lens 544/0 e 0 to 0 dl 1769083668 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 4798.982091] LustreError: 16652:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 20 previous similar messages [ 4807.705466] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 07:07:45 (1769083665) [ 4833.770351] LustreError: 40999:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 4844.807112] LustreError: 40999:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 4850.866712] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 07:08:28 (1769083708) [ 4876.711513] LustreError: 6532:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff97b27c3a6300 x1855013751555584/t0(0) o101->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:210/0 lens 576/0 e 0 to 0 dl 1769083745 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4876.728078] LustreError: 6532:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 4881.831654] LustreError: 6532:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4883.369901] LustreError: 6534:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 10000ms [ 4887.010796] LustreError: 42162:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff97b280794700 x1855013765731968/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:221/0 lens 544/0 e 0 to 0 dl 1769083756 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 4887.022928] LustreError: 42162:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 110 previous similar messages [ 4893.471289] LustreError: 6534:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4908.770310] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 07:09:26 (1769083766) [ 4998.833799] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 07:10:56 (1769083856) [ 5024.014137] LustreError: 10309:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff97b274aeb480 x1855013751632128/t0(0) o101->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:358/0 lens 576/0 e 0 to 0 dl 1769083893 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5024.032072] LustreError: 10309:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 108 previous similar messages [ 5024.045734] LustreError: 10309:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5024.471179] LustreError: 10309:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5040.284424] LustreError: 6536:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5040.295395] LustreError: 6536:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 5040.727304] LustreError: 6536:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5040.734121] LustreError: 6536:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 5056.043153] LustreError: 81679:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff97b27bc79f80 x1855013751650304/t0(0) o36->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:390/0 lens 504/0 e 0 to 0 dl 1769083925 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 5056.054408] LustreError: 81679:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 78 previous similar messages [ 5071.795385] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 07:12:09 (1769083929) [ 5101.425031] Lustre: DEBUG MARKER: phase 2 [ 5108.046753] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 07:12:45 (1769083965) [ 5184.548105] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 07:14:02 (1769084042) [ 5185.507580] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 5186.588857] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 07:14:04 (1769084044) [ 5189.127956] Lustre: DEBUG MARKER: Started rundbench load pid=133760 ... [ 5195.498459] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5197.455400] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 5198.801071] Lustre: Failing over lustre-MDT0000 [ 5198.929151] LustreError: 166110:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5198.935855] LustreError: 166110:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages [ 5199.014568] Lustre: server umount lustre-MDT0000 complete [ 5199.330187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5199.333231] LustreError: Skipped 3 previous similar messages [ 5199.335731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.342055] Lustre: Skipped 38 previous similar messages [ 5199.344769] LustreError: 7474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5199.352818] LustreError: 7474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 143 previous similar messages [ 5216.965110] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5216.968809] LDISKFS-fs (dm-0): recovery complete [ 5216.977303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5217.059524] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5217.067155] LustreError: Skipped 8 previous similar messages [ 5217.227182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5217.234302] Lustre: Skipped 7 previous similar messages [ 5217.259527] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5217.269726] Lustre: Skipped 11 previous similar messages [ 5218.413265] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5218.419336] Lustre: Skipped 11 previous similar messages [ 5219.833229] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5222.385352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5222.390676] Lustre: Skipped 25 previous similar messages [ 5222.423661] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5222.431211] Lustre: Skipped 11 previous similar messages [ 5222.464308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) [ 5222.470260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) [ 5225.380692] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5226.492203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5233.618963] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5235.714019] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 5236.757170] Lustre: Failing over lustre-MDT0001 [ 5236.765605] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.51@tcp (stopping) [ 5236.912238] Lustre: server umount lustre-MDT0001 complete [ 5254.720784] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 5254.722727] LDISKFS-fs (dm-1): recovery complete [ 5254.731616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5257.308068] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5261.278633] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:558 to 0x2c0000400:577) [ 5261.282991] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:575 to 0x280000400:609) [ 5263.495322] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5264.617932] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5272.138893] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5274.036436] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 5275.256153] Lustre: Failing over lustre-MDT0000 [ 5275.507242] Lustre: server umount lustre-MDT0000 complete [ 5292.526171] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5292.528177] LDISKFS-fs (dm-0): recovery complete [ 5292.532130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5294.594163] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5298.185675] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) [ 5298.185750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) [ 5300.216908] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5301.104335] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5308.154670] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5310.164843] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 5311.290422] Lustre: Failing over lustre-MDT0001 [ 5311.297404] LustreError: 168418:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0001-osc-MDT0001: can't precreate: rc = -5 [ 5311.306546] LustreError: 168418:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0001-osc-MDT0001: cannot precreate objects: rc = -5 [ 5311.323466] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.51@tcp (stopping) [ 5311.464145] Lustre: server umount lustre-MDT0001 complete [ 5328.337934] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5328.340578] LDISKFS-fs (dm-1): recovery complete [ 5328.353704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5330.812591] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5334.308967] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:691 to 0x280000400:737) [ 5334.308983] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:659 to 0x2c0000400:705) [ 5336.315515] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5337.268110] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5342.239496] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 07:16:39 (1769084199) [ 5466.797510] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5478.120378] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 5479.160841] Lustre: Failing over lustre-MDT0001 [ 5479.221493] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.51@tcp (stopping) [ 5482.464939] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5482.468692] Lustre: Skipped 2 previous similar messages [ 5485.154302] Lustre: server umount lustre-MDT0001 complete [ 5501.002173] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5501.005069] LDISKFS-fs (dm-1): recovery complete [ 5501.014054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5502.938532] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5508.337559] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1293 to 0x280000400:1313) [ 5508.337599] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1261 to 0x2c0000400:1281) [ 5509.885284] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5510.728249] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5635.997563] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5647.001681] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 5647.858357] Lustre: Failing over lustre-MDT0000 [ 5647.869563] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (stopping) [ 5647.873209] Lustre: Skipped 1 previous similar message [ 5654.118925] Lustre: server umount lustre-MDT0000 complete [ 5668.843072] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5668.844750] LDISKFS-fs (dm-0): recovery complete [ 5668.848344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5670.294395] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5675.880622] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6161 to 0x2c0000401:6177) [ 5675.881054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6224 to 0x280000401:6241) [ 5677.408178] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5678.131371] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5720.402195] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 07:22:58 (1769084578) [ 5843.693346] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5854.669910] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 5855.559923] Lustre: Failing over lustre-MDT0001 [ 5855.626729] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.51@tcp (stopping) [ 5855.629773] Lustre: Skipped 5 previous similar messages [ 5855.674709] LustreError: 176731:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5855.678561] LustreError: 176731:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 5855.750667] Lustre: server umount lustre-MDT0001 complete [ 5858.271588] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5858.275916] LustreError: Skipped 4 previous similar messages [ 5858.278037] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.283509] Lustre: Skipped 20 previous similar messages [ 5858.286044] LustreError: 7474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5858.293346] LustreError: 7474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 90 previous similar messages [ 5870.931681] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5870.933172] LDISKFS-fs (dm-1): recovery complete [ 5870.940274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5871.059983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5871.064053] Lustre: Skipped 5 previous similar messages [ 5871.073450] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5871.077372] Lustre: Skipped 5 previous similar messages [ 5872.774036] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5872.834545] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5872.837974] Lustre: Skipped 5 previous similar messages [ 5876.193784] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5876.195991] Lustre: Skipped 20 previous similar messages [ 5877.640111] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5877.643859] Lustre: Skipped 5 previous similar messages [ 5877.669253] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2194 to 0x2c0000400:2209) [ 5877.669907] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2227 to 0x280000400:2273) [ 5879.155743] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5879.847105] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6004.368281] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6015.207336] Lustre: DEBUG MARKER: test_70d fail mds2 2 times [ 6015.942215] Lustre: Failing over lustre-MDT0001 [ 6015.947887] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.51@tcp (stopping) [ 6016.266238] Lustre: server umount lustre-MDT0001 complete [ 6030.651086] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6030.652710] LDISKFS-fs (dm-1): recovery complete [ 6030.659044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6032.275906] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6037.352900] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2560 to 0x280000400:2593) [ 6037.353218] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2496 to 0x2c0000400:2529) [ 6038.739777] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6039.425502] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6043.028993] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 07:28:20 (1769084900) [ 6166.447701] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6177.184350] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 6177.831653] Lustre: Failing over lustre-MDT0000 [ 6177.982977] Lustre: server umount lustre-MDT0000 complete [ 6192.169178] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6192.170769] LDISKFS-fs (dm-0): recovery complete [ 6192.174437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6192.209937] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6192.213528] LustreError: Skipped 2 previous similar messages [ 6193.656151] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6200.341327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8594 to 0x2c0000401:8609) [ 6200.341839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8658 to 0x280000401:8673) [ 6201.744292] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6202.420301] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6327.097146] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6337.909256] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 6338.578829] Lustre: Failing over lustre-MDT0000 [ 6338.606634] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (stopping) [ 6338.609328] Lustre: Skipped 2 previous similar messages [ 6338.757629] Lustre: server umount lustre-MDT0000 complete [ 6352.856719] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6352.857922] LDISKFS-fs (dm-0): recovery complete [ 6352.860585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6354.350582] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6359.954945] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10330 to 0x2c0000401:10369) [ 6359.954958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10395 to 0x280000401:10433) [ 6361.267779] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6361.856360] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6365.459701] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 07:33:43 (1769085223) [ 6371.271981] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6372.852651] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 6373.557317] Lustre: Failing over lustre-OST0000 [ 6373.602145] Lustre: server umount lustre-OST0000 complete [ 6388.148554] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6388.150381] LDISKFS-fs (dm-2): recovery complete [ 6388.154685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6390.347397] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6393.881795] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6394.515084] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6403.710168] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6405.252735] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 6405.904274] Lustre: Failing over lustre-OST0000 [ 6405.959452] Lustre: server umount lustre-OST0000 complete [ 6420.081313] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6420.082882] LDISKFS-fs (dm-2): recovery complete [ 6420.086656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6421.952303] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6425.306399] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6425.967170] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6431.563642] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 07:34:49 (1769085289) [ 6554.629210] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6557.328705] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6568.106535] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 6568.768972] Lustre: Failing over lustre-MDT0000 [ 6568.819449] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.51@tcp (stopping) [ 6568.928151] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.931911] Lustre: Skipped 17 previous similar messages [ 6568.984122] LustreError: 188121:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6568.986333] LustreError: 188121:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 6569.028074] Lustre: server umount lustre-MDT0000 complete [ 6569.065212] LustreError: 8443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6569.071470] LustreError: 8443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 68 previous similar messages [ 6570.246781] LustreError: 42169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085428 with bad export cookie 14121417866464043423 [ 6570.247855] Lustre: Failing over lustre-MDT0001 [ 6570.252603] LustreError: 42169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6570.509806] Lustre: server umount lustre-MDT0001 complete [ 6585.170388] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6585.173128] LDISKFS-fs (dm-1): recovery complete [ 6585.179652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6585.206345] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6585.208524] LDISKFS-fs (dm-0): recovery complete [ 6585.213025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.272385] LustreError: 189419:0:(llog.c:1643:llog_backup()) MGC192.168.204.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6585.275363] Lustre: 189419:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6590.431261] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769085432/real 1769085432] req@ffff97b150fcaa00 x1855013790862720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769085448 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6590.437408] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6595.551419] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b150428000 x1855013790865664/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6595.626808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6595.629878] Lustre: Skipped 5 previous similar messages [ 6595.639078] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6595.641396] Lustre: Skipped 5 previous similar messages [ 6595.708570] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 6595.712489] LustreError: Skipped 10 previous similar messages [ 6597.084260] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6597.175421] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6597.177752] Lustre: Skipped 5 previous similar messages [ 6597.179613] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6601.185542] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6601.188690] Lustre: Skipped 17 previous similar messages [ 6601.769770] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6601.772240] Lustre: Skipped 5 previous similar messages [ 6601.789923] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2602 to 0x280000400:2625) [ 6601.791212] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2539 to 0x2c0000400:2561) [ 6606.295155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10492 to 0x280000401:10529) [ 6606.295473] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10429 to 0x2c0000401:10465) [ 6607.540516] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6608.120500] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6608.651179] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6732.818049] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6735.309093] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6746.053375] Lustre: DEBUG MARKER: fail mds1 mds2 2 times [ 6746.678815] Lustre: Failing over lustre-MDT0000 [ 6746.843032] Lustre: server umount lustre-MDT0000 complete [ 6748.037483] LustreError: 42169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085606 with bad export cookie 14121417866464938534 [ 6748.038511] Lustre: Failing over lustre-MDT0001 [ 6748.041424] LustreError: 42169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6748.184888] Lustre: server umount lustre-MDT0001 complete [ 6762.924411] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6762.926780] LDISKFS-fs (dm-1): recovery complete [ 6762.930658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6762.932236] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6762.937239] LDISKFS-fs (dm-0): recovery complete [ 6762.941608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6763.013197] LustreError: 192662:0:(llog.c:1643:llog_backup()) MGC192.168.204.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6763.016767] Lustre: 192662:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6765.535126] Lustre: 3666:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769085607/real 1769085607] req@ffff97b2460c9500 x1855013801079552/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769085623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6765.543349] Lustre: 3666:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 6773.727448] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b15109e680 x1855013801082880/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6775.176102] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6775.264535] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6779.955555] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2602 to 0x280000400:2657) [ 6779.955572] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2539 to 0x2c0000400:2593) [ 6784.602723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10492 to 0x280000401:10561) [ 6784.605271] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10429 to 0x2c0000401:10497) [ 6785.821874] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6786.378645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6786.886698] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6790.253957] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 07:40:48 (1769085648) [ 6792.685913] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6793.553453] Lustre: Failing over lustre-MDT0000 [ 6793.727455] Lustre: server umount lustre-MDT0000 complete [ 6807.786706] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6807.787890] LDISKFS-fs (dm-0): recovery complete [ 6807.791287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.826610] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6807.830218] LustreError: Skipped 3 previous similar messages [ 6809.136877] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6810.730611] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 6827.120612] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 6827.132206] Lustre: 192696:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b145b2d880 x1855013802393984/t339302416982(339302416982) o101->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:651/0 lens 520/3488 e 0 to 0 dl 1769085696 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6827.160440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10492 to 0x280000401:10593) [ 6827.160440] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10529) [ 6828.437365] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6828.964644] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6832.364780] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 07:41:30 (1769085690) [ 6835.004362] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6835.926609] Lustre: Failing over lustre-MDT0000 [ 6836.079958] Lustre: server umount lustre-MDT0000 complete [ 6850.217303] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6850.218787] LDISKFS-fs (dm-0): recovery complete [ 6850.221370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6851.531961] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6851.696575] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 6851.698049] LustreError: 192672:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b14b60ed80 x1855013802393984/t339302416982(339302416982) o101->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:675/0 lens 520/664 e 0 to 0 dl 1769085720 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6868.080158] Lustre: lustre-MDT0000: Client c24165d9-4026-4158-9006-fa0818e10e68 (at 192.168.204.51@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 6868.085174] Lustre: 192672:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b143cf2680 x1855013802393984/t339302416982(339302416982) o101->c24165d9-4026-4158-9006-fa0818e10e68@192.168.204.51@tcp:692/0 lens 520/3488 e 0 to 0 dl 1769085737 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6868.111572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10595 to 0x280000401:10625) [ 6868.111581] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10561) [ 6869.412747] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6869.939271] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6873.302313] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 07:42:11 (1769085731) [ 6874.682477] Lustre: Failing over lustre-OST0000 [ 6874.722495] Lustre: server umount lustre-OST0000 complete [ 6876.018504] Lustre: Failing over lustre-MDT0000 [ 6876.116116] Lustre: server umount lustre-MDT0000 complete [ 6888.910951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.394339] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6891.119836] Lustre: lustre-MDT0000: Denying connection for new client 6d6989fd-252d-42a2-b276-37a63759b65e (at 192.168.204.51@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 6894.068119] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10593) [ 6898.161450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6899.955418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10595 to 0x280000401:10657) [ 6900.141920] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6904.716361] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 07:42:42 (1769085762) [ 6905.053891] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6905.055473] LustreError: 192681:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b14b60c380 x1855013802694272/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:729/0 lens 1312/4320 e 0 to 0 dl 1769085774 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6907.612484] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6908.236227] Lustre: Failing over lustre-MDT0000 [ 6908.384284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6908.387098] Lustre: Skipped 7 previous similar messages [ 6908.447617] Lustre: server umount lustre-MDT0000 complete [ 6921.183132] Lustre: 192701:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769085763/real 1769085763] req@ffff97b14b60ce00 x1855013802694272/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1769085779 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6921.190694] Lustre: 192701:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 6922.577218] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6922.579161] LDISKFS-fs (dm-0): recovery complete [ 6922.583107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.906476] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6927.863561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:10689) [ 6927.863578] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10625) [ 6929.009600] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6929.529665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6933.043847] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 07:43:10 (1769085790) [ 6933.399208] LustreError: 192682:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b274b74380 x1855013802720512/t365072220172(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:2/0 lens 2520/4320 e 0 to 0 dl 1769085802 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6935.805833] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6938.325836] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6938.993102] Lustre: Failing over lustre-MDT0001 [ 6944.839266] Lustre: server umount lustre-MDT0001 complete [ 6958.703258] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6958.704465] LDISKFS-fs (dm-1): recovery complete [ 6958.707406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6959.976917] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6964.212291] Lustre: 192696:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b1443a7800 x1855013815724032/t34359740902(0) o36->6d6989fd-252d-42a2-b276-37a63759b65e@192.168.204.51@tcp:54/0 lens 560/2880 e 0 to 0 dl 1769085854 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6964.218399] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2604 to 0x2c0000400:2625) [ 6964.218412] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2668 to 0x280000400:2689) [ 6965.360908] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6965.910349] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6969.343991] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 07:43:47 (1769085827) [ 6969.684117] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6969.685676] Lustre: Skipped 1 previous similar message [ 6969.686895] LustreError: 192681:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b26f1cdc00 x1855013802753664/t365072220193(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:38/0 lens 2520/4320 e 0 to 0 dl 1769085838 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6972.052440] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6974.366338] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6974.951913] Lustre: Failing over lustre-MDT0000 [ 6975.223687] Lustre: server umount lustre-MDT0000 complete [ 6989.463325] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6989.464791] LDISKFS-fs (dm-0): recovery complete [ 6989.467632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6990.978391] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6994.960416] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10657) [ 6994.960426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:10721) [ 6996.044732] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6996.537135] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6998.418178] Lustre: Failing over lustre-MDT0001 [ 6998.510329] Lustre: server umount lustre-MDT0001 complete [ 7012.562781] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7012.563897] LDISKFS-fs (dm-1): recovery complete [ 7012.567755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7014.035352] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7017.971854] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2700 to 0x280000400:2721) [ 7017.971864] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2636 to 0x2c0000400:2657) [ 7019.061403] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7019.580695] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7023.220710] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 07:44:41 (1769085881) [ 7029.134069] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7031.615623] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7032.243191] Lustre: Failing over lustre-MDT0000 [ 7032.554968] Lustre: server umount lustre-MDT0000 complete [ 7033.756577] LustreError: 6519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769085891 with bad export cookie 14121417866465889638 [ 7033.757250] Lustre: Failing over lustre-MDT0001 [ 7033.760236] LustreError: 6519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7039.968222] LustreError: 208346:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff97b2702bed80 x1855013802814080/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7039.974740] LustreError: 208346:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 7039.977902] LustreError: 208346:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7040.076585] Lustre: server umount lustre-MDT0001 complete [ 7055.109452] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7055.112385] LDISKFS-fs (dm-1): recovery complete [ 7055.118617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7055.119356] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7055.123797] LDISKFS-fs (dm-0): recovery complete [ 7055.127976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7055.205287] LustreError: 209441:0:(llog.c:1643:llog_backup()) MGC192.168.204.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7055.207831] Lustre: 209441:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7058.911323] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b2702be300 x1855013802815232/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7060.285630] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7060.424427] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7065.223727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:10753) [ 7065.223730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10689) [ 7070.329464] Lustre: 210210:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b141f9c000 x1855013815791232/t42949673028(0) o36->6d6989fd-252d-42a2-b276-37a63759b65e@192.168.204.51@tcp:163/0 lens 560/2880 e 0 to 0 dl 1769085963 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7070.335669] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2732 to 0x280000400:2753) [ 7070.335679] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2668 to 0x2c0000400:2689) [ 7071.375844] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7071.865917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7072.300524] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7075.599065] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 07:45:33 (1769085933) [ 7075.940781] LustreError: 209457:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b2749d0700 x1855013815824384/t47244640324(0) o36->6d6989fd-252d-42a2-b276-37a63759b65e@192.168.204.51@tcp:144/0 lens 560/448 e 0 to 0 dl 1769085944 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7075.946934] LustreError: 209457:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 7081.260863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7081.926044] Lustre: Failing over lustre-MDT0000 [ 7082.287402] Lustre: server umount lustre-MDT0000 complete [ 7092.329222] Lustre: lustre-MDT0001: Client 6d6989fd-252d-42a2-b276-37a63759b65e (at 192.168.204.51@tcp) reconnecting [ 7092.332155] Lustre: Skipped 2 previous similar messages [ 7096.367414] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7096.369605] LDISKFS-fs (dm-0): recovery complete [ 7096.373114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7097.745543] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7101.947266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:10785) [ 7101.947306] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10721) [ 7102.948591] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7103.427091] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7106.728290] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 07:46:04 (1769085964) [ 7107.030368] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7107.031785] Lustre: Skipped 2 previous similar messages [ 7109.464189] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7110.067025] Lustre: Failing over lustre-MDT0001 [ 7110.149459] Lustre: server umount lustre-MDT0001 complete [ 7124.063847] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7124.065021] LDISKFS-fs (dm-1): recovery complete [ 7124.068728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7125.276024] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7129.584490] Lustre: 209456:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b2705ee300 x1855013815854464/t47244640393(0) o36->6d6989fd-252d-42a2-b276-37a63759b65e@192.168.204.51@tcp:198/0 lens 560/2880 e 0 to 0 dl 1769085998 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7129.590865] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2774 to 0x280000400:2817) [ 7129.590872] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2710 to 0x2c0000400:2753) [ 7129.591259] Lustre: 209456:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7130.797035] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7131.360507] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7134.993106] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 07:46:32 (1769085992) [ 7140.932896] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7143.595872] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7144.259383] Lustre: Failing over lustre-MDT0000 [ 7144.548312] Lustre: server umount lustre-MDT0000 complete [ 7158.501782] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7158.502911] LDISKFS-fs (dm-0): recovery complete [ 7158.505339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7159.959944] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7163.905398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:10817) [ 7163.905425] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10753) [ 7165.139663] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7165.731733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7167.685732] Lustre: Failing over lustre-MDT0001 [ 7167.772792] Lustre: server umount lustre-MDT0001 complete [ 7168.992024] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7168.996806] Lustre: Skipped 52 previous similar messages [ 7171.186757] LustreError: 210210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7171.191280] LustreError: 210210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 190 previous similar messages [ 7181.689933] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7181.690855] LDISKFS-fs (dm-1): recovery complete [ 7181.693567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7182.995363] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7186.935325] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2764 to 0x2c0000400:2785) [ 7186.935479] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2828 to 0x280000400:2849) [ 7188.042783] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7188.580476] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7192.244773] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 07:47:30 (1769086050) [ 7198.134745] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7201.001739] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7201.678231] Lustre: Failing over lustre-MDT0000 [ 7201.994767] LustreError: 218726:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7201.996827] LustreError: 218726:0:(obd_class.h:479:obd_check_dev()) Skipped 139 previous similar messages [ 7202.026164] Lustre: server umount lustre-MDT0000 complete [ 7202.272578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7202.275589] LustreError: Skipped 12 previous similar messages [ 7203.276825] LustreError: 6519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086061 with bad export cookie 14121417866465903274 [ 7203.277921] Lustre: Failing over lustre-MDT0001 [ 7203.282106] LustreError: 6519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7203.405466] Lustre: server umount lustre-MDT0001 complete [ 7218.325630] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7218.327795] LDISKFS-fs (dm-0): recovery complete [ 7218.332877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7218.403784] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7218.405672] LDISKFS-fs (dm-1): recovery complete [ 7218.409490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7223.775253] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769086065/real 1769086065] req@ffff97b281db9c00 x1855013802969088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769086081 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7223.785184] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7228.896717] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc3f95abb278e11c9 [ 7228.900412] Lustre: MGC192.168.204.151@tcp: Connection restored to 0@lo (at 0@lo) [ 7228.903151] Lustre: Skipped 58 previous similar messages [ 7229.021690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.024175] Lustre: Skipped 17 previous similar messages [ 7229.040639] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7229.042802] Lustre: Skipped 17 previous similar messages [ 7230.641935] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7230.770264] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7233.247762] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7233.250254] Lustre: Skipped 18 previous similar messages [ 7234.280161] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7234.284716] Lustre: Skipped 17 previous similar messages [ 7234.291714] Lustre: 220784:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b270567480 x1855013815920256/t55834574916(0) o36->6d6989fd-252d-42a2-b276-37a63759b65e@192.168.204.51@tcp:303/0 lens 560/2880 e 0 to 0 dl 1769086103 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7234.300551] Lustre: 220784:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7234.303116] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2860 to 0x280000400:2881) [ 7234.303277] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2796 to 0x2c0000400:2817) [ 7242.842318] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10785) [ 7242.842353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:10849) [ 7244.228242] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7244.767126] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7245.288881] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7248.885934] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 07:48:26 (1769086106) [ 7249.277085] LustreError: 8429:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b145197800 x1855013803003008/t386547056653(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:318/0 lens 1488/4320 e 0 to 0 dl 1769086118 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7249.288665] LustreError: 8429:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 3 previous similar messages [ 7251.858242] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7252.509657] Lustre: Failing over lustre-MDT0001 [ 7252.513414] LustreError: 221727:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x240003e99:0xa9:0x0].0x0 (ffff97b274cd7000) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7252.520203] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.51@tcp (stopping) [ 7252.522341] Lustre: Skipped 10 previous similar messages [ 7258.165205] Lustre: server umount lustre-MDT0001 complete [ 7272.453492] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7272.455048] LDISKFS-fs (dm-1): recovery complete [ 7272.458079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7273.931311] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7278.069637] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2828 to 0x2c0000400:2849) [ 7278.069637] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2892 to 0x280000400:2913) [ 7279.193659] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7279.744222] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7283.319698] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 07:49:01 (1769086141) [ 7286.282188] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7286.942123] Lustre: Failing over lustre-MDT0000 [ 7287.221300] Lustre: server umount lustre-MDT0000 complete [ 7301.422102] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7301.423773] LDISKFS-fs (dm-0): recovery complete [ 7301.426512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7302.847148] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7306.748476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:10881) [ 7306.748480] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10817) [ 7307.838440] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7308.336579] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7311.811064] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 07:49:29 (1769086169) [ 7314.792477] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7317.408974] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7318.035288] Lustre: Failing over lustre-MDT0000 [ 7318.346198] Lustre: server umount lustre-MDT0000 complete [ 7332.585377] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7332.586680] LDISKFS-fs (dm-0): recovery complete [ 7332.589282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.012149] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7337.987112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:10913) [ 7337.987158] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10849) [ 7339.132502] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7339.698764] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7341.682224] Lustre: Failing over lustre-MDT0001 [ 7341.770723] Lustre: server umount lustre-MDT0001 complete [ 7355.949356] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7355.951095] LDISKFS-fs (dm-1): recovery complete [ 7355.955730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7357.363367] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7361.529572] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2892 to 0x280000400:2945) [ 7361.529624] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2828 to 0x2c0000400:2881) [ 7362.709572] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7363.286210] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7366.789176] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 07:50:24 (1769086224) [ 7367.160108] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7367.161481] Lustre: Skipped 5 previous similar messages [ 7369.649485] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7372.172925] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7372.820197] Lustre: Failing over lustre-MDT0000 [ 7373.077254] Lustre: server umount lustre-MDT0000 complete [ 7374.268617] LustreError: 145084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086232 with bad export cookie 14121417866465913641 [ 7374.270168] Lustre: Failing over lustre-MDT0001 [ 7374.272104] LustreError: 145084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7379.936227] LustreError: 229526:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff97b2778cca80 x1855013803111168/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7379.945706] LustreError: 229526:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7380.046676] Lustre: server umount lustre-MDT0001 complete [ 7395.094368] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7395.096696] LDISKFS-fs (dm-0): recovery complete [ 7395.097865] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7395.100977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7395.101353] LDISKFS-fs (dm-1): recovery complete [ 7395.104483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7395.202105] LustreError: 230627:0:(llog.c:1643:llog_backup()) MGC192.168.204.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7395.205888] Lustre: 230627:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7399.391454] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff97b27b712680 x1855013803112320/t0(0) o250->MGC192.168.204.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7399.399228] LustreError: 3663:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [ 7401.035071] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7401.200726] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7405.620385] Lustre: 230641:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff97b145c34380 x1855013815998080/t68719476738(0) o36->6d6989fd-252d-42a2-b276-37a63759b65e@192.168.204.51@tcp:499/0 lens 496/2888 e 0 to 0 dl 1769086299 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 7405.626369] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2892 to 0x280000400:2977) [ 7405.628786] Lustre: 230641:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7405.628956] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2828 to 0x2c0000400:2913) [ 7405.650825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:10945) [ 7405.652726] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10881) [ 7406.931319] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7407.523307] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7408.096190] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7411.548683] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 07:51:09 (1769086269) [ 7414.644771] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7415.268977] Lustre: Failing over lustre-MDT0000 [ 7415.521569] Lustre: server umount lustre-MDT0000 complete [ 7428.201300] Lustre: lustre-MDT0001: Client 6d6989fd-252d-42a2-b276-37a63759b65e (at 192.168.204.51@tcp) reconnecting [ 7428.204419] Lustre: Skipped 1 previous similar message [ 7429.645302] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7429.646568] LDISKFS-fs (dm-0): recovery complete [ 7429.649356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7429.683120] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7429.686938] LustreError: Skipped 11 previous similar messages [ 7431.052665] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7435.261462] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10913) [ 7435.261466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:10977) [ 7436.388456] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7436.899969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7440.414104] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 07:51:38 (1769086298) [ 7443.214254] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7443.867628] Lustre: Failing over lustre-MDT0001 [ 7443.966677] Lustre: server umount lustre-MDT0001 complete [ 7458.158388] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7458.159694] LDISKFS-fs (dm-1): recovery complete [ 7458.163955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7459.616826] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7463.413485] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2892 to 0x280000400:3009) [ 7463.413485] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2828 to 0x2c0000400:2945) [ 7464.654838] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7465.201421] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7468.758642] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 07:52:06 (1769086326) [ 7471.741870] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7474.324795] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7474.952994] Lustre: Failing over lustre-MDT0000 [ 7475.163412] Lustre: server umount lustre-MDT0000 complete [ 7489.228769] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7489.230498] LDISKFS-fs (dm-0): recovery complete [ 7489.232995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7490.661314] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7494.654432] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10945) [ 7494.654436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:11009) [ 7495.827773] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7496.379881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7498.293770] Lustre: Failing over lustre-MDT0001 [ 7498.382620] Lustre: server umount lustre-MDT0001 complete [ 7512.454017] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7512.455388] LDISKFS-fs (dm-1): recovery complete [ 7512.459119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7513.775012] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7517.687188] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2892 to 0x280000400:3041) [ 7517.687191] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2828 to 0x2c0000400:2977) [ 7518.911606] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7519.507774] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7523.150710] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 07:53:01 (1769086381) [ 7523.537189] LustreError: 231634:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff97b2705cd180 x1855013816052096/t81604378626(0) o36->6d6989fd-252d-42a2-b276-37a63759b65e@192.168.204.51@tcp:592/0 lens 496/456 e 0 to 0 dl 1769086392 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 7523.546367] LustreError: 231634:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 6 previous similar messages [ 7526.017445] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7528.697952] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7529.386863] Lustre: Failing over lustre-MDT0000 [ 7529.562688] Lustre: server umount lustre-MDT0000 complete [ 7530.829766] LustreError: 6518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086388 with bad export cookie 14121417866465921152 [ 7530.831328] Lustre: Failing over lustre-MDT0001 [ 7530.833237] LustreError: 6518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7530.937450] Lustre: server umount lustre-MDT0001 complete [ 7545.762437] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7545.762546] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7545.764464] LDISKFS-fs (dm-1): recovery complete [ 7545.765944] LDISKFS-fs (dm-0): recovery complete [ 7545.767567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7545.771594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7545.859554] LustreError: 241250:0:(llog.c:1643:llog_backup()) MGC192.168.204.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7545.862507] Lustre: 241250:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7557.070585] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7557.203144] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7561.784621] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2828 to 0x2c0000400:3009) [ 7561.784664] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2892 to 0x280000400:3073) [ 7561.808263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10499 to 0x2c0000401:10977) [ 7561.808351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10659 to 0x280000401:11041) [ 7563.096851] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7563.676930] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7564.202019] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7567.784751] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 07:53:45 (1769086425) [ 7568.455496] Lustre: 242489:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 6d6989fd-252d-42a2-b276-37a63759b65e at adminstrative request [ 7573.338812] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 07:53:51 (1769086431) [ 7575.416257] Lustre: Failing over lustre-MDT0000 [ 7575.700398] Lustre: server umount lustre-MDT0000 complete [ 7588.529755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.979134] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7594.004886] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11028 to 0x2c0000401:11073) [ 7594.004909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11093 to 0x280000401:11137) [ 7595.192100] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7595.759544] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7599.326666] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 07:54:17 (1769086457) [ 7603.307616] Lustre: Failing over lustre-OST0000 [ 7605.381130] Lustre: server umount lustre-OST0000 complete [ 7618.143047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7620.127917] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7623.309227] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7623.920522] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7627.605464] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 07:54:45 (1769086485) [ 7628.920075] Lustre: Failing over lustre-MDT0000 [ 7629.135884] Lustre: server umount lustre-MDT0000 complete [ 7631.620931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.032036] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7633.758625] Lustre: lustre-MDT0000: Denying connection for new client 17d98c2b-a1c3-46ff-8ac4-38a06bad2756 (at 192.168.204.51@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7633.764427] Lustre: Skipped 2 previous similar messages [ 7636.982203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11093 to 0x280000401:11169) [ 7636.982288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11028 to 0x2c0000401:11105) [ 7641.313233] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 07:54:59 (1769086499) [ 7644.129324] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7645.021210] Lustre: Failing over lustre-OST0000 [ 7645.053180] Lustre: server umount lustre-OST0000 complete [ 7659.015918] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7659.017784] LDISKFS-fs (dm-2): recovery complete [ 7659.022107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7660.930990] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7663.952450] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7664.489249] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7667.963265] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 07:55:25 (1769086525) [ 7670.708174] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7672.577853] Lustre: Failing over lustre-OST0000 [ 7672.609904] Lustre: server umount lustre-OST0000 complete [ 7686.729555] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7686.731101] LDISKFS-fs (dm-2): recovery complete [ 7686.734670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7688.784388] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7688.843075] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.51@tcp inode [0x2000301a1:0x5:0x0] object 0x280000401:11171 extent [0-4194303]: client csum a3df3bb0, server csum c618b086 [ 7691.952245] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7692.526631] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7695.969367] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 07:55:53 (1769086553) [ 7698.517968] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7700.902688] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7702.882641] Lustre: Failing over lustre-MDT0000 [ 7703.173413] Lustre: server umount lustre-MDT0000 complete [ 7704.446634] LustreError: 145084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086562 with bad export cookie 14121417866465947920 [ 7704.449300] Lustre: Failing over lustre-OST0000 [ 7704.450195] LustreError: 145084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7704.490471] Lustre: server umount lustre-OST0000 complete [ 7718.966656] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7718.968019] LDISKFS-fs (dm-0): recovery complete [ 7718.971088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7729.120590] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc3f95abb278ec013 [ 7730.508581] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7734.352303] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11028 to 0x2c0000401:11137) [ 7744.866218] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7744.867499] LDISKFS-fs (dm-2): recovery complete [ 7744.870854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7746.879192] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7752.609640] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 07:56:50 (1769086610) [ 7757.472402] Lustre: Failing over lustre-OST0000 [ 7757.541266] Lustre: server umount lustre-OST0000 complete [ 7759.061136] Lustre: Failing over lustre-MDT0000 [ 7759.423947] Lustre: server umount lustre-MDT0000 complete [ 7771.241105] LustreError: 241268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.246864] LustreError: 241268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 244 previous similar messages [ 7772.466709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7773.931450] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7777.786605] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11028 to 0x2c0000401:11169) [ 7782.628219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7784.557629] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7785.320145] Lustre: lustre-OST0000: Denying connection for new client 808732f4-8413-4fcd-a9aa-da53ba398312 (at 192.168.204.51@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 7785.329769] Lustre: Skipped 1 previous similar message [ 7790.697529] Lustre: lustre-OST0000: Denying connection for new client 808732f4-8413-4fcd-a9aa-da53ba398312 (at 192.168.204.51@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 7790.702652] Lustre: Skipped 1 previous similar message [ 7800.937863] Lustre: lustre-OST0000: Denying connection for new client 808732f4-8413-4fcd-a9aa-da53ba398312 (at 192.168.204.51@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 7800.945195] Lustre: Skipped 1 previous similar message [ 7821.417419] Lustre: lustre-OST0000: Denying connection for new client 808732f4-8413-4fcd-a9aa-da53ba398312 (at 192.168.204.51@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 7821.425027] Lustre: Skipped 3 previous similar messages [ 7853.500288] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 7853.503636] Lustre: 255531:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 17d98c2b-a1c3-46ff-8ac4-38a06bad2756@ [ 7853.510250] Lustre: 255531:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 7853.515023] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 7853.522226] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7853.522359] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 7853.525895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11220 to 0x280000401:11241) [ 7853.529455] Lustre: Skipped 69 previous similar messages [ 7853.547188] Lustre: Skipped 21 previous similar messages [ 7857.976888] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 7862.373845] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 7864.352181] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 07:58:42 (1769086722) [ 7865.977943] Lustre: Failing over lustre-OST0000 [ 7866.000202] LustreError: 256770:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7866.001960] LustreError: 256770:0:(obd_class.h:479:obd_check_dev()) Skipped 163 previous similar messages [ 7866.018746] Lustre: server umount lustre-OST0000 complete [ 7868.895586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7868.895625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.898016] LustreError: Skipped 17 previous similar messages [ 7868.903251] Lustre: Skipped 64 previous similar messages [ 7879.247727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7879.309964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7879.313124] Lustre: Skipped 22 previous similar messages [ 7879.317120] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 7879.319644] Lustre: Skipped 22 previous similar messages [ 7880.619440] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7880.621808] Lustre: Skipped 21 previous similar messages [ 7881.268048] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7885.199615] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 07:59:03 (1769086743) [ 7886.490136] Lustre: Failing over lustre-OST0000 [ 7886.529977] Lustre: server umount lustre-OST0000 complete [ 7899.349402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7900.724633] LustreError: 258512:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 7900.727543] LustreError: 258512:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 78 previous similar messages [ 7901.226780] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7906.783154] Lustre: *** cfs_fail_loc=715, val=40*** [ 7915.999384] Lustre: 3663:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769086758/real 1769086758] req@ffff97b26e61f800 x1855013803494272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1769086774 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 7916.016173] Lustre: 3663:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 29 previous similar messages [ 7916.022564] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:25 [ 7916.028618] Lustre: Skipped 1 previous similar message [ 7916.597664] Lustre: lustre-OST0000: Client 808732f4-8413-4fcd-a9aa-da53ba398312 (at 192.168.204.51@tcp) reconnected, waiting for 3 clients in recovery for 1:24 [ 7922.143163] Lustre: *** cfs_fail_loc=715, val=40*** [ 7922.146714] Lustre: Skipped 2 previous similar messages [ 7932.383627] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:09 [ 7932.386708] Lustre: Skipped 1 previous similar message [ 7938.527116] Lustre: *** cfs_fail_loc=715, val=40*** [ 7938.528440] Lustre: Skipped 2 previous similar messages [ 7940.775148] LustreError: 258512:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7940.781145] LustreError: 258512:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 78 previous similar messages [ 7942.358031] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7942.884136] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7946.399885] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 08:00:04 (1769086804) [ 7947.724497] Lustre: Failing over lustre-MDT0000 [ 7947.745719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7947.749094] Lustre: Skipped 19 previous similar messages [ 7947.981180] Lustre: server umount lustre-MDT0000 complete [ 7961.215469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7963.111190] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7966.697641] LustreError: 259955:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 7969.759327] Lustre: *** cfs_fail_loc=715, val=80*** [ 7969.762245] Lustre: Skipped 2 previous similar messages [ 7980.137636] Lustre: lustre-MDT0000: Client 808732f4-8413-4fcd-a9aa-da53ba398312 (at 192.168.204.51@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 7980.140647] Lustre: Skipped 1 previous similar message [ 7982.048533] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7986.655359] Lustre: *** cfs_fail_loc=715, val=80*** [ 7986.657915] Lustre: Skipped 1 previous similar message [ 7996.521177] Lustre: lustre-MDT0000: Client 808732f4-8413-4fcd-a9aa-da53ba398312 (at 192.168.204.51@tcp) reconnected, waiting for 2 clients in recovery for 1:10 [ 7998.431477] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8003.039154] Lustre: *** cfs_fail_loc=715, val=80*** [ 8003.040336] Lustre: Skipped 1 previous similar message [ 8011.881326] Lustre: lustre-MDT0000: Client 808732f4-8413-4fcd-a9aa-da53ba398312 (at 192.168.204.51@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 8014.815538] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8020.959190] Lustre: *** cfs_fail_loc=715, val=80*** [ 8020.961756] Lustre: Skipped 2 previous similar messages [ 8028.265390] Lustre: lustre-MDT0000: Client 808732f4-8413-4fcd-a9aa-da53ba398312 (at 192.168.204.51@tcp) reconnected, waiting for 2 clients in recovery for 0:39 [ 8030.175490] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8046.559613] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8046.783141] LustreError: 259955:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8046.788495] Lustre: 259955:0:(ldlm_lib.c:2931:target_recovery_thread()) too long recovery - read logs [ 8046.790985] LustreError: dumping log to /tmp/lustre-log.1769086904.259955 [ 8046.851736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11255 to 0x280000401:11273) [ 8046.851810] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11182 to 0x2c0000401:11201) [ 8048.121568] Lustre: DEBUG MARKER: oleg451-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8048.642089] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8052.238831] Lustre: DEBUG MARKER: == replay-single test complete, duration 7726 sec ======== 08:01:50 (1769086910) [ 8052.746147] Lustre: DEBUG MARKER: === replay-single: start cleanup 08:01:50 (1769086910) === [ 8055.265474] Lustre: DEBUG MARKER: === replay-single: finish cleanup 08:01:53 (1769086913) === [ 8092.083921] Lustre: server umount lustre-MDT0000 complete [ 8094.764806] LustreError: 145084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769086952 with bad export cookie 14121417866465960569 [ 8094.766273] LustreError: MGC192.168.204.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8094.767995] LustreError: 145084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8094.770883] LustreError: Skipped 7 previous similar messages [ 8094.873828] Lustre: server umount lustre-MDT0001 complete [ 8107.839402] Lustre: server umount lustre-OST0000 complete [ 8121.040408] Lustre: server umount lustre-OST0001 complete [ 8128.256198] Lustre: DEBUG MARKER: oleg451-server.virtnet: executing unload_modules_local [ 8129.714543] Key type lgssc unregistered [ 8129.895418] LNet: 263127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8129.900527] LNetError: 263127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8129.913502] LNet: Removed LNI 192.168.204.151@tcp [ 8130.368205] Key type .llcrypt unregistered [ 8130.370196] Key type ._llcrypt unregistered