[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494905694 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003394] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010005] pid_max: default: 32768 minimum: 301 [ 0.011133] LSM: Security Framework initializing [ 0.012071] Yama: becoming mindful. [ 0.013055] SELinux: Initializing. [ 0.015054] *** VALIDATE selinux *** [ 0.024700] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029701] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032117] *** VALIDATE tmpfs *** [ 0.034483] *** VALIDATE proc *** [ 0.036125] *** VALIDATE cgroup *** [ 0.037011] *** VALIDATE cgroup2 *** [ 0.039042] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040130] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042030] Spectre V2 : User space: Vulnerable [ 0.043010] Speculative Store Bypass: Vulnerable [ 0.045923] debug: unmapping init [mem 0xffffffffb5c59000-0xffffffffb5c60fff] [ 0.047944] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048714] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049020] ... version: 2 [ 0.050013] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056278] rcu: Hierarchical SRCU implementation. [ 0.058611] smp: Bringing up secondary CPUs ... [ 0.059615] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.063425] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.248327] node 0 deferred pages initialised in 180ms [ 0.251329] devtmpfs: initialized [ 0.252228] x86/mm: Memory block size: 128MB [ 0.254895] gcov: version magic: 0x41383552 [ 0.256104] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.257082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.258273] pinctrl core: initialized pinctrl subsystem [ 0.259185] [ 0.259758] ************************************************************* [ 0.260014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261013] ** ** [ 0.262018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.263016] ** ** [ 0.264016] ** This means that this kernel is built to expose internal ** [ 0.265014] ** IOMMU data structures, which may compromise security on ** [ 0.266023] ** your system. ** [ 0.267014] ** ** [ 0.268016] ** If you see this message and you are not debugging the ** [ 0.269014] ** kernel, report this immediately to your vendor! ** [ 0.270015] ** ** [ 0.271016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272015] ************************************************************* [ 0.273751] NET: Registered protocol family 16 [ 0.274462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.275048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.276061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.277446] cpuidle: using governor menu [ 0.279729] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.282454] PCI: Using configuration type 1 for base access [ 0.284140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.294077] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.295018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.297067] cryptd: max_cpu_qlen set to 1000 [ 0.301045] ACPI: Added _OSI(Module Device) [ 0.302025] ACPI: Added _OSI(Processor Device) [ 0.304019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.307017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.314000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.320419] ACPI: Interpreter enabled [ 0.323076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.325025] ACPI: Using IOAPIC for interrupt routing [ 0.328111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.333301] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.344027] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.346071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.349033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.353131] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.358542] acpiphp: Slot [2] registered [ 0.360206] acpiphp: Slot [5] registered [ 0.362214] acpiphp: Slot [6] registered [ 0.364200] acpiphp: Slot [7] registered [ 0.365201] acpiphp: Slot [8] registered [ 0.367205] acpiphp: Slot [9] registered [ 0.369212] acpiphp: Slot [10] registered [ 0.370122] acpiphp: Slot [3] registered [ 0.372121] acpiphp: Slot [4] registered [ 0.374126] acpiphp: Slot [11] registered [ 0.375104] acpiphp: Slot [12] registered [ 0.377104] acpiphp: Slot [13] registered [ 0.378111] acpiphp: Slot [14] registered [ 0.379103] acpiphp: Slot [15] registered [ 0.381131] acpiphp: Slot [16] registered [ 0.383111] acpiphp: Slot [17] registered [ 0.384086] acpiphp: Slot [18] registered [ 0.385127] acpiphp: Slot [19] registered [ 0.387112] acpiphp: Slot [20] registered [ 0.388110] acpiphp: Slot [21] registered [ 0.390106] acpiphp: Slot [22] registered [ 0.391125] acpiphp: Slot [23] registered [ 0.393106] acpiphp: Slot [24] registered [ 0.395174] acpiphp: Slot [25] registered [ 0.396114] acpiphp: Slot [26] registered [ 0.398136] acpiphp: Slot [27] registered [ 0.400172] acpiphp: Slot [28] registered [ 0.401097] acpiphp: Slot [29] registered [ 0.403153] acpiphp: Slot [30] registered [ 0.404145] acpiphp: Slot [31] registered [ 0.406103] PCI host bridge to bus 0000:00 [ 0.408031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.412042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.414038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.417040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.420046] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.423048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.425222] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.428298] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.432591] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.443015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.447058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.450031] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.453032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.456029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.460426] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.463015] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.466073] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.468837] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.474015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.486797] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.491025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.497124] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.506025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.514030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.533026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.545048] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.553016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.562015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.579021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.590000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.597030] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.604025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.625024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.636482] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.643013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.649023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.665013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.677014] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.683020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.690024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.715022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.735427] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.741014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.749016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.763018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.772417] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.774380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.777359] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.779386] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.782398] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.787153] iommu: Default domain type: Passthrough [ 0.788427] SCSI subsystem initialized [ 0.790116] ACPI: bus type USB registered [ 0.792153] usbcore: registered new interface driver usbfs [ 0.795112] usbcore: registered new interface driver hub [ 0.798091] usbcore: registered new device driver usb [ 0.801195] pps_core: LinuxPPS API ver. 1 registered [ 0.803014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.808065] PTP clock support registered [ 0.811116] EDAC MC: Ver: 3.0.0 [ 0.813142] PCI: Using ACPI for IRQ routing [ 0.816931] NetLabel: Initializing [ 0.818071] NetLabel: domain hash size = 128 [ 0.821015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.824150] NetLabel: unlabeled traffic allowed by default [ 0.827301] vgaarb: loaded [ 0.829229] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.831014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.839000] clocksource: Switched to clocksource kvm-clock [ 0.947562] VFS: Disk quotas dquot_6.6.0 [ 0.949204] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.951896] *** VALIDATE ramfs *** [ 0.953341] *** VALIDATE hugetlbfs *** [ 0.955712] pnp: PnP ACPI init [ 0.958288] pnp: PnP ACPI: found 6 devices [ 0.973832] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.976984] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.979416] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.982155] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.984774] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.987581] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.990558] NET: Registered protocol family 2 [ 0.993073] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.020445] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.025766] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.031359] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.034705] TCP: Hash tables configured (established 65536 bind 65536) [ 1.037506] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.040836] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.044246] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.047724] NET: Registered protocol family 1 [ 1.050630] RPC: Registered named UNIX socket transport module. [ 1.053317] RPC: Registered udp transport module. [ 1.055396] RPC: Registered tcp transport module. [ 1.056912] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.060445] NET: Registered protocol family 44 [ 1.062139] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.064886] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.067309] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.069688] PCI: CLS 0 bytes, default 64 [ 1.071208] Unpacking initramfs... [ 2.515646] debug: unmapping init [mem 0xffff8e3f3cc54000-0xffff8e3f3ffbffff] [ 2.520129] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.522653] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.525907] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.012645] Initialise system trusted keyrings [ 3.014079] Key type blacklist registered [ 3.015870] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.025069] zbud: loaded [ 3.028333] *** VALIDATE nfs *** [ 3.029716] *** VALIDATE nfs4 *** [ 3.031471] pstore: using deflate compression [ 3.034059] Platform Keyring initialized [ 3.142666] NET: Registered protocol family 38 [ 3.144216] Key type asymmetric registered [ 3.145906] Asymmetric key parser 'x509' registered [ 3.147895] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.151424] io scheduler mq-deadline registered [ 3.153048] io scheduler kyber registered [ 3.154631] io scheduler bfq registered [ 3.157107] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.160511] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.163552] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.166641] ACPI: Power Button [PWRF] [ 3.173334] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.180967] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.195796] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.203635] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.226148] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.256637] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.285456] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.291568] Non-volatile memory driver v1.3 [ 3.292939] Linux agpgart interface v0.103 [ 3.329274] virtio_blk virtio1: [vda] 135992 512-byte logical blocks (69.6 MB/66.4 MiB) [ 3.332125] vda: detected capacity change from 0 to 69627904 [ 3.347461] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.349748] vdb: detected capacity change from 0 to 1073741824 [ 3.364498] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.367476] vdc: detected capacity change from 0 to 2621440000 [ 3.385248] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.387733] vdd: detected capacity change from 0 to 2621440000 [ 3.402874] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.405504] vde: detected capacity change from 0 to 4294967296 [ 3.419601] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.422017] vdf: detected capacity change from 0 to 4294967296 [ 3.437504] libphy: Fixed MDIO Bus: probed [ 3.443320] usbcore: registered new interface driver usbserial_generic [ 3.445470] usbserial: USB Serial support registered for generic [ 3.447489] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.451673] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.453394] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.455797] mousedev: PS/2 mouse device common for all mice [ 3.459657] rtc_cmos 00:05: RTC can wake from S4 [ 3.463210] rtc_cmos 00:05: registered as rtc0 [ 3.465379] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.468928] intel_pstate: CPU model not supported [ 3.472906] hid: raw HID events driver (C) Jiri Kosina [ 3.473198] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.477489] usbcore: registered new interface driver usbhid [ 3.477495] usbhid: USB HID core driver [ 3.477665] drop_monitor: Initializing network drop monitor service [ 3.477813] Initializing XFRM netlink socket [ 3.479275] NET: Registered protocol family 10 [ 3.488282] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.489861] Segment Routing with IPv6 [ 3.493512] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.497702] NET: Registered protocol family 17 [ 3.499228] mpls_gso: MPLS GSO support [ 3.519204] RAS: Correctable Errors collector initialized. [ 3.521271] AVX version of gcm_enc/dec engaged. [ 3.522833] AES CTR mode by8 optimization enabled [ 3.612711] sched_clock: Marking stable (3612623647, 0)->(4634193053, -1021569406) [ 3.616912] registered taskstats version 1 [ 3.619552] Loading compiled-in X.509 certificates [ 3.621278] zswap: loaded using pool lzo/zbud [ 3.672954] Key type big_key registered [ 3.686861] Key type encrypted registered [ 3.688972] ima: No TPM chip found, activating TPM-bypass! [ 3.691554] ima: Allocated hash algorithm: sha1 [ 3.693662] ima: No architecture policies found [ 3.695851] evm: Initialising EVM extended attributes: [ 3.698089] evm: security.selinux [ 3.699489] evm: security.ima [ 3.700674] evm: security.capability [ 3.702206] evm: HMAC attrs: 0x1 [ 3.705224] rtc_cmos 00:05: setting system clock to 2026-06-07 17:09:08 UTC (1780852148) [ 3.711926] debug: unmapping init [mem 0xffffffffb6c03000-0xffffffffb6dfffff] [ 3.715364] debug: unmapping init [mem 0xffffffffb5982000-0xffffffffb5c58fff] [ 3.724078] Write protecting the kernel read-only data: 28672k [ 3.727794] debug: unmapping init [mem 0xffffffffb4003000-0xffffffffb41fffff] [ 3.730954] debug: unmapping init [mem 0xffffffffb4914000-0xffffffffb49fffff] [ 3.772670] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.780794] systemd[1]: Detected virtualization kvm. [ 3.783028] systemd[1]: Detected architecture x86-64. [ 3.785262] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.815570] systemd[1]: No hostname configured. [ 3.817571] systemd[1]: Set hostname to . [ 3.819912] random: systemd: uninitialized urandom read (16 bytes read) [ 3.822669] systemd[1]: Initializing machine ID from random generator. [ 3.860373] random: ln: uninitialized urandom read (6 bytes read) [ 3.956621] random: systemd: uninitialized urandom read (16 bytes read) [ 3.959667] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.963826] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.969600] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.622064] device-mapper: uevent: version 1.0.3 [ 4.624093] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.300080] random: fast init done [ 5.322691] virtio_net virtio0 ens2: renamed from eth0 [ 5.370404] scsi host0: ata_piix [ 5.389073] scsi host1: ata_piix [ 5.391106] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.393789] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.928620] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.210407] random: crng init done [ 10.211990] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.561615] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.706176] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.973965] SELinux: Disabled at runtime. [ 12.039360] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.053582] systemd[1]: Detected virtualization kvm. [ 12.058794] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.584298] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.587860] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.593948] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.598201] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.601610] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.607922] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.612608] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkey[ 12.667084] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS gen.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.059193] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.311671] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.378211] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.465271] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.487214] EDAC sbridge: Ver: 1.1.2 [ 15.206598] Key type dns_resolver registered [ 15.506100] NFS: Registering the id_resolver key type [ 15.508105] Key type id_resolver registered [ 15.509654] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg307-server login: [ 57.720029] hrtimer: interrupt took 6021600 ns [ 58.155537] libcfs: loading out-of-tree module taints kernel. [ 58.188410] Key type ._llcrypt registered [ 58.190713] Key type .llcrypt registered [ 58.288811] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 77.091560] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 78.695142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 78.722172] alg: No test for adler32 (adler32-zlib) [ 80.203040] Lustre: Lustre: Build Version: 2.17.53_57_g73f2eba [ 81.181764] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 82.953535] Key type lgssc registered [ 84.768244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.885144] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 147.279214] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 161.637478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.663811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.055050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.142470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.304516] Lustre: lustre-MDT0000: new disk, initializing [ 163.471561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.507750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.209570] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.189055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 183.297293] Lustre: 6501:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 183.323513] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 183.329190] Lustre: Skipped 1 previous similar message [ 183.419573] Lustre: lustre-MDT0001: new disk, initializing [ 183.481114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 183.505877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 183.512891] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.656675] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.647690] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 202.187682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.423902] Lustre: lustre-OST0000: new disk, initializing [ 202.426854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.431584] Lustre: 8405:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.548527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.895844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.904497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.977555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.236715] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.796067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.893480] Lustre: lustre-OST0001: new disk, initializing [ 223.897270] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 223.901409] Lustre: 9461:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.952836] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 229.824901] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.999233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 232.009244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.068650] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 242.200172] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.590385] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 255.746902] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing check_logdir /tmp/testlogs/ [ 260.189720] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing yml_node [ 266.605460] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 269.601505] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 272.894564] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 274.800249] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sun Jun 7 13:13:37 EDT 2026 [ 293.223384] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 295.507761] Lustre: DEBUG MARKER: === replay-single: start setup 13:13:58 (1780852438) === [ 299.515567] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing check_config_client /mnt/lustre [ 324.998819] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 329.825634] Lustre: 13173:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 334.591681] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 340.174881] Lustre: DEBUG MARKER: === replay-single: finish setup 13:14:43 (1780852483) === [ 341.948570] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 13:14:45 (1780852485) [ 349.584557] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 351.660179] Lustre: Failing over lustre-MDT0000 [ 351.967167] Lustre: server umount lustre-MDT0000 complete [ 352.759701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 352.779945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.298001] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.312922] Lustre: Skipped 1 previous similar message [ 359.793054] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.820443] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 360.432758] LustreError: 7820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.451384] LustreError: 7820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 364.904850] LustreError: 10435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.028304] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.054607] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 371.679293] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780852500/real 1780852500] req@ffff8e3e8389ca80 x1867358902800000/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780852516 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 371.745294] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 375.133856] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 375.137520] LDISKFS-fs (dm-0): recovery complete [ 375.149213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.165387] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.189797] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 381.937445] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9f77dbd [ 381.947869] Lustre: MGC192.168.203.107@tcp: Connection restored to 0@lo (at 0@lo) [ 382.268545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.607819] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 387.565518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 387.572372] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 387.633546] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 396.188227] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 397.709995] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 407.297696] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 13:15:50 (1780852550) [ 409.737979] Lustre: Failing over lustre-OST0000 [ 409.932939] Lustre: server umount lustre-OST0000 complete [ 412.644055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 412.656312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 412.672310] Lustre: Skipped 1 previous similar message [ 412.685949] LustreError: 8389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 412.713768] LustreError: 8389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 428.574539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 428.771902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.196299] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 430.709214] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 430.713190] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 430.740403] Lustre: Skipped 3 previous similar messages [ 435.168254] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 442.939584] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 444.584378] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 453.911769] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 13:16:37 (1780852597) [ 461.740686] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 464.200127] Lustre: Failing over lustre-MDT0000 [ 464.356119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 464.361187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.368628] Lustre: Skipped 1 previous similar message [ 464.375488] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.386879] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 464.460254] Lustre: server umount lustre-MDT0000 complete [ 485.345455] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780852614/real 1780852614] req@ffff8e3e82c3d500 x1867358902864000/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780852630 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 485.380948] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 487.939986] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 487.948262] LDISKFS-fs (dm-0): recovery complete [ 487.958512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.598824] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9f7894e [ 495.614123] Lustre: MGC192.168.203.107@tcp: Connection restored to 0@lo (at 0@lo) [ 495.621130] Lustre: Skipped 1 previous similar message [ 495.878606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 500.091456] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 501.216642] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 501.224949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 502.330791] Lustre: lustre-MDT0000: Denying connection for new client db1c0470-ab22-49f3-abc1-6bb6ef0b41ea (at 192.168.203.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 507.755776] Lustre: lustre-MDT0000: Denying connection for new client db1c0470-ab22-49f3-abc1-6bb6ef0b41ea (at 192.168.203.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 512.872097] Lustre: lustre-MDT0000: Denying connection for new client db1c0470-ab22-49f3-abc1-6bb6ef0b41ea (at 192.168.203.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 517.987900] Lustre: lustre-MDT0000: Denying connection for new client db1c0470-ab22-49f3-abc1-6bb6ef0b41ea (at 192.168.203.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 523.106944] Lustre: lustre-MDT0000: Denying connection for new client db1c0470-ab22-49f3-abc1-6bb6ef0b41ea (at 192.168.203.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 533.347425] Lustre: lustre-MDT0000: Denying connection for new client db1c0470-ab22-49f3-abc1-6bb6ef0b41ea (at 192.168.203.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 533.368976] Lustre: Skipped 1 previous similar message [ 553.827697] Lustre: lustre-MDT0000: Denying connection for new client db1c0470-ab22-49f3-abc1-6bb6ef0b41ea (at 192.168.203.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:17 [ 553.851377] Lustre: Skipped 3 previous similar messages [ 571.501771] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 571.513343] Lustre: 17837:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client dbac4332-c95c-43c5-a5a9-376b724e3ba4@ [ 571.535783] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 571.562668] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 571.566346] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 571.570617] Lustre: Skipped 2 previous similar messages [ 571.582855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 571.583194] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 584.856661] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 13:18:47 (1780852727) [ 594.170408] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 596.409749] Lustre: Failing over lustre-MDT0000 [ 596.753984] Lustre: server umount lustre-MDT0000 complete [ 597.473873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 597.480862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.496587] Lustre: Skipped 3 previous similar messages [ 597.504062] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.537140] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 614.879150] Lustre: 3655:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780852743/real 1780852743] req@ffff8e3f854e3b80 x1867358902923904/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780852759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 614.915329] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.988965] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 618.992586] LDISKFS-fs (dm-0): recovery complete [ 619.002218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.316122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.359571] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 629.539277] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 630.765120] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 630.775249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 632.206537] Lustre: lustre-MDT0000: Denying connection for new client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 632.241473] Lustre: Skipped 3 previous similar messages [ 698.722810] Lustre: lustre-MDT0000: Denying connection for new client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:01 [ 698.750402] Lustre: Skipped 12 previous similar messages [ 700.500262] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 700.506607] Lustre: 19546:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client db1c0470-ab22-49f3-abc1-6bb6ef0b41ea@ [ 700.532798] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 700.561597] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 700.564686] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 700.595477] Lustre: Skipped 2 previous similar messages [ 700.599714] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 700.600497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 713.723728] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 13:20:56 (1780852856) [ 722.641071] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 725.570358] Lustre: Failing over lustre-MDT0000 [ 725.881543] Lustre: server umount lustre-MDT0000 complete [ 726.498513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 726.504833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 726.521786] Lustre: Skipped 3 previous similar messages [ 726.533125] LustreError: 7820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 726.552976] LustreError: 7820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 743.394199] Lustre: 3654:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780852872/real 1780852872] req@ffff8e3fbc2b5f80 x1867358902982144/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780852888 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 743.414847] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 748.454370] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 748.458252] LDISKFS-fs (dm-0): recovery complete [ 748.465483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.635759] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3fbf8e6300 x1867358902990208/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 754.090373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 754.181705] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 756.069880] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 759.406158] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 759.481790] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 759.482538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 759.709484] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 768.541925] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 770.837302] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 781.404758] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 13:22:04 (1780852924) [ 789.956526] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 792.559578] Lustre: Failing over lustre-MDT0000 [ 792.973424] Lustre: server umount lustre-MDT0000 complete [ 795.103948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 795.118701] Lustre: Skipped 5 previous similar messages [ 811.488168] Lustre: 3654:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780852939/real 1780852939] req@ffff8e3fb62ece00 x1867358903019008/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780852955 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 811.522411] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 816.002643] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 816.004618] LDISKFS-fs (dm-0): recovery complete [ 816.016294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 821.737625] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3fbf892680 x1867358903028480/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 822.000072] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 822.637125] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 825.535186] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 827.379672] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 827.387532] Lustre: Skipped 4 previous similar messages [ 827.477865] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 827.513703] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 827.520321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 832.755071] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 834.513485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 842.491697] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 13:23:05 (1780852985) [ 850.636652] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 852.608096] Lustre: Failing over lustre-MDT0000 [ 852.888175] Lustre: server umount lustre-MDT0000 complete [ 852.961211] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.979670] Lustre: Skipped 3 previous similar messages [ 858.085226] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.107591] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 70 previous similar messages [ 873.312843] Lustre: 3653:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853002/real 1780853002] req@ffff8e3e838de680 x1867358903053824/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780853018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 873.344972] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 875.044373] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 875.046941] LDISKFS-fs (dm-0): recovery complete [ 875.059210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 883.971946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 883.979599] Lustre: Skipped 1 previous similar message [ 884.015963] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 885.120554] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 889.118206] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 889.483967] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 889.524872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:193) [ 889.534231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 895.964983] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 897.573222] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 905.049745] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 13:24:08 (1780853048) [ 912.411267] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 914.719321] Lustre: Failing over lustre-MDT0000 [ 914.914969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 914.921219] Lustre: Skipped 2 previous similar messages [ 915.072270] Lustre: server umount lustre-MDT0000 complete [ 935.889111] Lustre: 3653:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853064/real 1780853064] req@ffff8e3fbc2b7100 x1867358903087744/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780853080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 935.927222] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 939.761824] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 939.766062] LDISKFS-fs (dm-0): recovery complete [ 939.774445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.599093] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 948.234311] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 950.533826] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 951.869768] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 951.910607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 951.911184] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 957.221457] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 958.872691] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 967.076443] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 13:25:10 (1780853110) [ 973.259233] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 974.970050] Lustre: Failing over lustre-MDT0000 [ 975.249538] Lustre: server umount lustre-MDT0000 complete [ 977.376597] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 977.381903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 977.400075] Lustre: Skipped 7 previous similar messages [ 993.768775] Lustre: 3654:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853122/real 1780853122] req@ffff8e3e832e3b80 x1867358903120512/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780853138 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 993.785650] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 998.661747] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 998.663622] LDISKFS-fs (dm-0): recovery complete [ 998.682854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.002312] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9f7a6eb [ 1004.013691] Lustre: MGC192.168.203.107@tcp: Connection restored to 0@lo (at 0@lo) [ 1004.021992] Lustre: Skipped 11 previous similar messages [ 1004.272910] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1007.822415] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1009.722942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1009.728345] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1014.305667] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1015.627509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1022.825629] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 13:26:06 (1780853166) [ 1023.528602] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1023.530979] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1023.535413] LustreError: 6511:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3fb87f9c00 x1867358883956608/t38654705666(0) o35->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:704/0 lens 392/456 e 0 to 0 dl 1780853179 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1031.666113] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1033.827623] Lustre: Failing over lustre-MDT0000 [ 1034.099292] Lustre: server umount lustre-MDT0000 complete [ 1035.232992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1054.616788] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1054.622089] LDISKFS-fs (dm-0): recovery complete [ 1054.635504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.856145] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3e82366680 x1867358903163008/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1062.158748] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1062.246176] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1062.260146] Lustre: Skipped 1 previous similar message [ 1066.560894] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1067.573237] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1067.581349] Lustre: Skipped 1 previous similar message [ 1067.586100] Lustre: 6511:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3fb8505500 x1867358883956608/t38654705666(0) o35->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:748/0 lens 392/456 e 0 to 0 dl 1780853223 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1067.642429] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1067.649043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1074.081315] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1075.815651] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1083.592547] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 13:27:07 (1780853227) [ 1090.455297] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1092.722735] Lustre: Failing over lustre-MDT0000 [ 1092.925185] Lustre: server umount lustre-MDT0000 complete [ 1109.472503] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853237/real 1780853237] req@ffff8e3facfb8380 x1867358903188864/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780853253 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1109.499579] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1109.520226] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1109.528602] LustreError: Skipped 1 previous similar message [ 1114.482668] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.522503] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 149 previous similar messages [ 1115.041821] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1115.043597] LDISKFS-fs (dm-0): recovery complete [ 1115.051285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.712710] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3fb85f5f80 x1867358903197568/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1119.972731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1123.777314] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1125.477359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1125.480834] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1130.290200] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1131.907494] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1139.417693] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 13:28:02 (1780853282) [ 1146.856602] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1147.785324] Lustre: *** cfs_fail_loc=114, val=0*** [ 1150.506351] Lustre: Failing over lustre-MDT0000 [ 1150.732786] Lustre: server umount lustre-MDT0000 complete [ 1150.950675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1150.953558] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.971260] Lustre: Skipped 11 previous similar messages [ 1172.554980] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1172.556855] LDISKFS-fs (dm-0): recovery complete [ 1172.564710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1181.665642] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3fb638ca80 x1867358903231104/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1181.834910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.842834] Lustre: Skipped 4 previous similar messages [ 1184.739582] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1187.356442] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1187.357774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1190.007665] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1191.305760] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1199.195822] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 13:29:02 (1780853342) [ 1206.704844] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1207.820804] Lustre: *** cfs_fail_loc=128, val=0*** [ 1210.714368] Lustre: Failing over lustre-MDT0000 [ 1211.148684] Lustre: server umount lustre-MDT0000 complete [ 1232.340729] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1232.343747] LDISKFS-fs (dm-0): recovery complete [ 1232.349610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.754236] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1239.767939] Lustre: Skipped 1 previous similar message [ 1241.038669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1241.047737] Lustre: Skipped 2 previous similar messages [ 1243.787978] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1245.226345] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1245.234755] Lustre: Skipped 2 previous similar messages [ 1245.263701] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1245.263732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1250.523554] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1252.132697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1258.937638] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 13:30:02 (1780853402) [ 1265.780434] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1267.825283] Lustre: Failing over lustre-MDT0000 [ 1268.018220] Lustre: server umount lustre-MDT0000 complete [ 1270.754092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1270.764267] LustreError: Skipped 1 previous similar message [ 1287.137684] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853415/real 1780853415] req@ffff8e3f91a5c700 x1867358903288960/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780853431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1287.153109] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1287.159451] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.167226] LustreError: Skipped 2 previous similar messages [ 1287.582271] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1287.586292] LDISKFS-fs (dm-0): recovery complete [ 1287.596956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.352147] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3e832c1f80 x1867358903297408/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1300.824406] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1302.002807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1302.006386] Lustre: Skipped 20 previous similar messages [ 1302.314445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1302.315559] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1307.720837] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1309.022643] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1315.813898] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 13:30:59 (1780853459) [ 1321.876813] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1323.509866] Lustre: Failing over lustre-MDT0000 [ 1323.696537] Lustre: server umount lustre-MDT0000 complete [ 1344.339130] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1344.343047] LDISKFS-fs (dm-0): recovery complete [ 1344.362212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.540796] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1359.530535] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1359.530965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1364.477708] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1365.930326] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1374.443781] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 13:31:57 (1780853517) [ 1381.002772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1389.788945] Lustre: Failing over lustre-MDT0000 [ 1390.107393] Lustre: server umount lustre-MDT0000 complete [ 1412.872927] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1412.877526] LDISKFS-fs (dm-0): recovery complete [ 1412.895880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.671967] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3fb8504000 x1867358903369984/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1417.310731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1417.328813] Lustre: Skipped 2 previous similar messages [ 1421.650915] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1425.770203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1425.774557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1429.333293] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1430.899734] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1456.820928] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 13:33:20 (1780853600) [ 1464.239139] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1466.376826] Lustre: Failing over lustre-MDT0000 [ 1466.590607] Lustre: server umount lustre-MDT0000 complete [ 1466.855305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1466.864129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.875637] Lustre: Skipped 17 previous similar messages [ 1486.806608] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1486.808715] LDISKFS-fs (dm-0): recovery complete [ 1486.813366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.540665] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1500.769173] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1500.770042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1504.505379] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1505.762876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1515.549334] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 13:34:18 (1780853658) [ 1523.143762] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1525.487914] Lustre: Failing over lustre-MDT0000 [ 1525.967274] Lustre: server umount lustre-MDT0000 complete [ 1548.141096] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1548.151816] LDISKFS-fs (dm-0): recovery complete [ 1548.164381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.694425] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1554.699273] Lustre: Skipped 4 previous similar messages [ 1556.805577] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1558.595679] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1558.609701] Lustre: Skipped 4 previous similar messages [ 1558.651792] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1558.652894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1563.779990] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1565.210722] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1573.315631] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 13:35:16 (1780853716) [ 1581.484029] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1583.540219] Lustre: Failing over lustre-MDT0000 [ 1583.950489] Lustre: server umount lustre-MDT0000 complete [ 1599.455580] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780853728/real 1780853728] req@ffff8e3e83346d80 x1867358903530624/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780853744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1599.489797] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 1599.512516] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1599.532265] LustreError: Skipped 4 previous similar messages [ 1605.420800] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1605.424303] LDISKFS-fs (dm-0): recovery complete [ 1605.437632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.752387] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1615.386452] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1615.386622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1619.928357] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1621.320437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1629.369655] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 13:36:12 (1780853772) [ 1636.290412] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1638.272241] Lustre: Failing over lustre-MDT0000 [ 1638.579145] Lustre: server umount lustre-MDT0000 complete [ 1640.928743] LustreError: 9755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.960198] LustreError: 9755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 283 previous similar messages [ 1661.270978] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1661.273257] LDISKFS-fs (dm-0): recovery complete [ 1661.290925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.551741] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3fb8d47b80 x1867358903571456/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1671.676794] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1673.371197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1673.377435] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1678.747510] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1680.312940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1687.746923] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 13:37:11 (1780853831) [ 1694.623827] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1697.333551] Lustre: Failing over lustre-MDT0000 [ 1697.820293] Lustre: server umount lustre-MDT0000 complete [ 1723.982898] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1723.986214] LDISKFS-fs (dm-0): recovery complete [ 1723.992722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.626856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.634794] Lustre: Skipped 8 previous similar messages [ 1724.669582] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1724.686855] Lustre: Skipped 4 previous similar messages [ 1729.241363] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1730.134700] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1730.134936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1737.697654] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1739.654106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1748.191943] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 13:38:11 (1780853891) [ 1754.571287] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1756.492083] Lustre: Failing over lustre-MDT0000 [ 1756.541965] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 1756.551904] Lustre: Skipped 1 previous similar message [ 1756.742183] Lustre: server umount lustre-MDT0000 complete [ 1760.740368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1760.754447] LustreError: Skipped 2 previous similar messages [ 1778.490604] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1778.494619] LDISKFS-fs (dm-0): recovery complete [ 1778.500835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.336749] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3fb875c700 x1867358903640064/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1790.991624] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1792.084101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1792.085346] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1797.957708] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1799.787616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1807.404180] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 13:39:10 (1780853950) [ 1814.406657] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1816.267648] Lustre: Failing over lustre-MDT0000 [ 1816.473913] Lustre: server umount lustre-MDT0000 complete [ 1838.541175] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1838.543419] LDISKFS-fs (dm-0): recovery complete [ 1838.552583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.037211] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1849.834744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1849.838020] Lustre: Skipped 35 previous similar messages [ 1849.989284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1849.989302] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1856.004983] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1857.486381] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1863.945519] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 13:40:07 (1780854007) [ 1870.760538] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1872.511254] Lustre: Failing over lustre-MDT0000 [ 1872.744488] Lustre: server umount lustre-MDT0000 complete [ 1893.607514] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1893.612148] LDISKFS-fs (dm-0): recovery complete [ 1893.629512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.284078] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1907.804079] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1907.812068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1915.538676] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1917.432694] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1926.061676] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 13:41:08 (1780854068) [ 1934.044170] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1935.932945] Lustre: Failing over lustre-MDT0000 [ 1938.248691] Lustre: server umount lustre-MDT0000 complete [ 1961.118216] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1961.120991] LDISKFS-fs (dm-0): recovery complete [ 1961.132757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.025351] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3fbc32d880 x1867358903742976/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1969.382467] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1970.765766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1970.766112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1976.381373] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1978.133178] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1985.822984] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 13:42:09 (1780854129) [ 1991.725281] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1993.631930] Lustre: Failing over lustre-MDT0000 [ 1993.908652] Lustre: server umount lustre-MDT0000 complete [ 1996.256306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.270186] Lustre: Skipped 37 previous similar messages [ 2016.859367] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2016.861819] LDISKFS-fs (dm-0): recovery complete [ 2016.878590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.578909] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2028.115384] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 2028.122070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 2032.887911] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2034.185615] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2041.718644] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 13:43:04 (1780854184) [ 2048.262353] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2050.095674] Lustre: Failing over lustre-MDT0000 [ 2050.305249] Lustre: server umount lustre-MDT0000 complete [ 2071.450928] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2071.456037] LDISKFS-fs (dm-0): recovery complete [ 2071.466213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.780517] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2081.787156] Lustre: Skipped 8 previous similar messages [ 2084.190640] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2085.961789] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2085.971404] Lustre: Skipped 8 previous similar messages [ 2085.994569] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2085.997708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2091.005071] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2092.524447] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2099.460157] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 13:44:02 (1780854242) [ 2106.354651] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2108.253435] Lustre: Failing over lustre-MDT0000 [ 2108.551573] Lustre: server umount lustre-MDT0000 complete [ 2127.842633] Lustre: 3655:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780854256/real 1780854256] req@ffff8e3f85441500 x1867358903836928/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780854272 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2127.890431] Lustre: 3655:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2127.902820] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2127.926650] LustreError: Skipped 8 previous similar messages [ 2131.719687] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2131.722441] LDISKFS-fs (dm-0): recovery complete [ 2131.728556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2143.191786] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2143.873416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 2143.873753] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 2150.573719] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2152.092419] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2161.148481] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 13:45:04 (1780854304) [ 2169.414651] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2171.152682] Lustre: Failing over lustre-MDT0000 [ 2171.362595] Lustre: server umount lustre-MDT0000 complete [ 2195.153590] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2195.155517] LDISKFS-fs (dm-0): recovery complete [ 2195.166953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2201.068055] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9f8a919 [ 2205.789372] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2206.838980] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 2206.838980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:993) [ 2212.835751] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2214.225738] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2223.598899] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 13:46:06 (1780854366) [ 2231.247172] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2233.162583] Lustre: Failing over lustre-MDT0000 [ 2233.450862] Lustre: server umount lustre-MDT0000 complete [ 2242.528232] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2242.553609] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 345 previous similar messages [ 2255.868127] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2255.872766] LDISKFS-fs (dm-0): recovery complete [ 2255.884241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.033849] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9f8aee5 [ 2264.392553] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2264.402776] Lustre: Skipped 8 previous similar messages [ 2269.854911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2269.855087] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2269.934663] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2278.726969] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2280.474463] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2289.410427] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 13:47:12 (1780854432) [ 2298.392941] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2300.676515] Lustre: Failing over lustre-MDT0000 [ 2300.896777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2300.914871] LustreError: Skipped 4 previous similar messages [ 2301.204499] Lustre: server umount lustre-MDT0000 complete [ 2325.527526] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2325.530352] LDISKFS-fs (dm-0): recovery complete [ 2325.536605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.127668] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3facedd500 x1867358903957376/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2332.154866] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [ 2332.381190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.384408] Lustre: Skipped 9 previous similar messages [ 2336.095313] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2337.963287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2337.963858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2343.516541] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2345.124769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2353.861984] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 13:48:16 (1780854496) [ 2361.059700] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2363.546296] Lustre: Failing over lustre-MDT0000 [ 2363.756589] Lustre: server umount lustre-MDT0000 complete [ 2385.781432] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2385.783915] LDISKFS-fs (dm-0): recovery complete [ 2385.789638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2393.073452] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9f8bb1e [ 2397.318741] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2398.811214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2398.811308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2404.197641] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2405.671664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2413.100590] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 13:49:16 (1780854556) [ 2417.494580] Lustre: 71729:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 3b8e3256-46d0-40e0-9570-180f14e2d182 at adminstrative request [ 2423.459192] Lustre: Failing over lustre-MDT0000 [ 2423.686618] Lustre: server umount lustre-MDT0000 complete [ 2442.450409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.671576] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2456.561432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2456.577333] Lustre: Skipped 42 previous similar messages [ 2456.636997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2456.637919] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2463.475593] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2465.405966] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2470.475438] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2480.181271] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2487.543839] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 13:50:30 (1780854630) [ 2488.641386] Lustre: 73625:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 3b8e3256-46d0-40e0-9570-180f14e2d182 at adminstrative request [ 2499.441620] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 13:50:42 (1780854642) [ 2506.733217] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2508.660197] Lustre: Failing over lustre-MDT0000 [ 2508.814585] Lustre: server umount lustre-MDT0000 complete [ 2529.482414] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2529.484962] LDISKFS-fs (dm-0): recovery complete [ 2529.492656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.894945] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2545.335191] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2545.345951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2551.877440] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2553.905740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2562.581978] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 13:51:45 (1780854705) [ 2570.027971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2571.962623] Lustre: Failing over lustre-MDT0000 [ 2572.280687] Lustre: server umount lustre-MDT0000 complete [ 2594.715295] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2594.722405] LDISKFS-fs (dm-0): recovery complete [ 2594.739112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.248385] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2608.148024] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2608.148997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2613.312329] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2614.877276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2623.070668] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 13:52:46 (1780854766) [ 2629.986838] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2631.759026] Lustre: Failing over lustre-MDT0000 [ 2631.984438] Lustre: server umount lustre-MDT0000 complete [ 2633.696794] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.709754] Lustre: Skipped 40 previous similar messages [ 2652.659728] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2652.667384] LDISKFS-fs (dm-0): recovery complete [ 2652.677481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.310037] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9f8d0fe [ 2662.899585] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2665.023793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2665.028175] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2669.444296] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2670.890642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2678.540120] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 13:53:41 (1780854821) [ 2685.252754] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2686.960667] Lustre: Failing over lustre-MDT0000 [ 2687.208619] Lustre: server umount lustre-MDT0000 complete [ 2708.390920] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2708.394503] LDISKFS-fs (dm-0): recovery complete [ 2708.400599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.057483] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2718.068193] Lustre: Skipped 9 previous similar messages [ 2721.767763] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2722.347071] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2722.361779] Lustre: Skipped 9 previous similar messages [ 2722.405285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2722.407839] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2731.237318] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2733.007528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2742.289303] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 13:54:44 (1780854884) [ 2751.651940] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2753.733917] Lustre: Failing over lustre-MDT0000 [ 2754.026178] Lustre: server umount lustre-MDT0000 complete [ 2773.471114] Lustre: 3654:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780854902/real 1780854902] req@ffff8e3fad2fea00 x1867358904213760/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780854918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2773.492702] Lustre: 3654:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 2773.507056] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2773.528214] LustreError: Skipped 9 previous similar messages [ 2775.147646] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2775.150161] LDISKFS-fs (dm-0): recovery complete [ 2775.162527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.730245] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9f8dbcb [ 2787.656269] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2789.455042] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 2789.457997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 2794.658218] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2796.188598] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2803.921447] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 13:55:47 (1780854947) [ 2810.934888] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2813.114380] Lustre: Failing over lustre-MDT0000 [ 2813.367790] Lustre: server umount lustre-MDT0000 complete [ 2835.779691] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2835.783482] LDISKFS-fs (dm-0): recovery complete [ 2835.791156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.938563] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2846.358523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2846.364607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2852.690464] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2854.274711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2863.287978] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 13:56:46 (1780855006) [ 2871.718457] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2874.116263] Lustre: Failing over lustre-MDT0000 [ 2874.554720] Lustre: server umount lustre-MDT0000 complete [ 2876.775046] LustreError: 10435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.794673] LustreError: 10435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 327 previous similar messages [ 2897.059239] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2897.062183] LDISKFS-fs (dm-0): recovery complete [ 2897.067177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.496387] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3f8740fb80 x1867358904295680/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2902.515275] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 101 previous similar messages [ 2902.808127] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2902.814869] Lustre: Skipped 9 previous similar messages [ 2907.415815] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2908.286317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 2908.287056] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 2914.386428] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2915.910912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2924.316604] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 13:57:47 (1780855067) [ 2931.659580] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2933.906630] Lustre: Failing over lustre-MDT0000 [ 2934.234397] Lustre: server umount lustre-MDT0000 complete [ 2938.854881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2938.867929] LustreError: Skipped 5 previous similar messages [ 2958.081065] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2958.083157] LDISKFS-fs (dm-0): recovery complete [ 2958.101940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.677614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2964.681980] Lustre: Skipped 9 previous similar messages [ 2969.408492] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2970.214388] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 2970.220152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 2977.221164] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2979.255881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2987.652623] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 13:58:50 (1780855130) [ 2994.883898] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2996.878081] Lustre: Failing over lustre-MDT0000 [ 2997.159726] Lustre: server umount lustre-MDT0000 complete [ 3018.176200] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3018.178303] LDISKFS-fs (dm-0): recovery complete [ 3018.189982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.172371] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3032.629623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3032.630339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3038.523419] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3040.156170] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3047.500551] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 13:59:50 (1780855190) [ 3054.871510] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3056.569905] Lustre: Failing over lustre-MDT0000 [ 3056.787403] Lustre: server umount lustre-MDT0000 complete [ 3079.423946] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3079.427089] LDISKFS-fs (dm-0): recovery complete [ 3079.434858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.891733] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3088.876463] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3088.888257] Lustre: Skipped 41 previous similar messages [ 3088.964574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3088.965623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 3095.346508] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3097.149559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3105.578822] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 14:00:48 (1780855248) [ 3112.279664] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3113.930404] Lustre: Failing over lustre-MDT0000 [ 3114.192798] Lustre: server umount lustre-MDT0000 complete [ 3134.169326] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3134.171769] LDISKFS-fs (dm-0): recovery complete [ 3134.178062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3144.166681] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9f8ff57 [ 3148.296416] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3149.893406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 3149.895307] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 3154.836927] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3156.462292] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3164.618829] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 14:01:47 (1780855307) [ 3165.676401] Lustre: 93971:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 3b8e3256-46d0-40e0-9570-180f14e2d182 at adminstrative request [ 3173.830222] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 14:01:57 (1780855317) [ 3179.744196] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3181.396382] Lustre: Failing over lustre-MDT0000 [ 3181.581546] Lustre: server umount lustre-MDT0000 complete [ 3193.618151] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3193.620948] LDISKFS-fs (dm-0): recovery complete [ 3193.631549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.079685] Lustre: lustre-MDT0000: Aborting client recovery [ 3194.082938] LustreError: 95252:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3194.085797] LustreError: 95283:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0000-osd: get update log duration 0, retries 0, failed: rc = -108 [ 3194.088520] Lustre: 95285:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3194.099918] Lustre: 95285:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 3b8e3256-46d0-40e0-9570-180f14e2d182@ [ 3194.113711] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3194.123574] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3194.131074] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3194.207695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1480 to 0x280000401:1505) [ 3194.218709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1479 to 0x2c0000401:1505) [ 3198.365285] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3199.464927] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3218.752190] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 14:02:42 (1780855362) [ 3226.146431] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3227.995797] Lustre: Failing over lustre-MDT0000 [ 3228.012740] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 3228.234626] Lustre: server umount lustre-MDT0000 complete [ 3239.935795] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3239.939250] LDISKFS-fs (dm-0): recovery complete [ 3239.955534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3240.357116] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3240.388924] Lustre: lustre-MDT0000: Aborting client recovery [ 3240.391206] LustreError: 97100:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3240.397694] Lustre: 97133:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3240.417830] Lustre: 97133:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 3240.423188] Lustre: 97133:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3240.434567] Lustre: 97133:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3240.444256] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3240.455631] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3240.478516] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3240.530651] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 3240.533310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 3245.128601] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3245.542741] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3258.802844] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3258.804920] Lustre: Skipped 1 previous similar message [ 3265.657967] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 14:03:28 (1780855408) [ 3272.840499] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3274.393366] Lustre: Failing over lustre-MDT0000 [ 3274.745661] Lustre: server umount lustre-MDT0000 complete [ 3276.295231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.315048] Lustre: Skipped 43 previous similar messages [ 3286.034530] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3286.039708] LDISKFS-fs (dm-0): recovery complete [ 3286.050261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.399923] Lustre: lustre-MDT0000: Aborting client recovery [ 3286.402055] LustreError: 98949:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3286.403482] Lustre: 98983:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3286.415594] Lustre: 98983:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 3286.426899] Lustre: 98983:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3286.432367] Lustre: 98983:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3286.435711] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3286.450853] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3286.468983] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3286.516280] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 3286.523638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 3290.131463] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3291.639622] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3308.920543] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 14:04:12 (1780855452) [ 3310.016510] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3310.024453] LustreError: 6508:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3e82c3d500 x1867358885041152/t201863462916(0) o36->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:725/0 lens 512/456 e 0 to 0 dl 1780855465 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3314.244479] Lustre: Failing over lustre-MDT0000 [ 3314.636939] Lustre: server umount lustre-MDT0000 complete [ 3323.605806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.977453] Lustre: lustre-MDT0000: Aborting client recovery [ 3323.981768] LustreError: 100257:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3323.987503] Lustre: 100290:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3323.996197] Lustre: 100290:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 3324.001953] Lustre: 100290:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3324.009596] Lustre: 100290:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3324.016460] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3324.039110] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3324.074868] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3324.169521] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 3324.175539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1601) [ 3328.194855] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3329.017514] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3348.357851] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3349.960227] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 14:04:53 (1780855493) [ 3357.406859] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3360.098781] Lustre: Failing over lustre-MDT0000 [ 3360.375931] Lustre: server umount lustre-MDT0000 complete [ 3371.958245] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3371.961455] LDISKFS-fs (dm-0): recovery complete [ 3371.971334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3372.369534] Lustre: lustre-MDT0000: Aborting client recovery [ 3372.372311] LustreError: 102205:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3372.378487] Lustre: 102238:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3372.389373] Lustre: 102238:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 3372.401259] Lustre: 102238:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3372.407809] Lustre: 102238:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3372.413086] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3372.423352] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3372.434911] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3372.473121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1633) [ 3372.473418] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 3376.185959] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3377.635893] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3394.856799] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 14:05:38 (1780855538) [ 3427.021899] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3428.756156] Lustre: Failing over lustre-MDT0000 [ 3428.833936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3428.845442] Lustre: Skipped 3 previous similar messages [ 3429.058791] Lustre: server umount lustre-MDT0000 complete [ 3449.268402] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3449.270523] LDISKFS-fs (dm-0): recovery complete [ 3449.276326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.312883] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780855578/real 1780855578] req@ffff8e3facd92a00 x1867358904695424/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780855594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3449.348711] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 3449.356477] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3449.365331] LustreError: Skipped 11 previous similar messages [ 3453.125203] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3454.306624] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3454.318644] Lustre: Skipped 7 previous similar messages [ 3455.056905] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3455.063431] Lustre: Skipped 7 previous similar messages [ 3455.087913] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3455.090893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3459.252712] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3460.786863] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3478.337983] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 14:07:01 (1780855621) [ 3499.962234] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3509.209783] Lustre: Failing over lustre-MDT0000 [ 3509.455523] Lustre: server umount lustre-MDT0000 complete [ 3510.634583] LustreError: 9755:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3510.650215] LustreError: 9755:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 228 previous similar messages [ 3529.484843] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3529.487978] LDISKFS-fs (dm-0): recovery complete [ 3529.498936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3537.891069] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3fb4bd5180 x1867358904897280/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3537.908181] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 13 previous similar messages [ 3538.127256] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3538.135035] Lustre: Skipped 20 previous similar messages [ 3541.362361] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3546.258696] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3546.259734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3549.318718] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3550.593684] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3569.490298] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 14:08:32 (1780855712) [ 3571.415177] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3572.318495] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3572.322708] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3577.424175] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 14:08:40 (1780855720) [ 3600.616208] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3610.538992] Lustre: Failing over lustre-OST0000 [ 3610.602729] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3610.607588] LustreError: Skipped 8 previous similar messages [ 3610.623701] Lustre: server umount lustre-OST0000 complete [ 3629.046953] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3629.050648] LDISKFS-fs (dm-2): recovery complete [ 3629.064128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3629.185684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.188396] Lustre: Skipped 10 previous similar messages [ 3632.497445] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3686.375735] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 14:10:30 (1780855830) [ 3691.847307] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3693.747288] Lustre: Failing over lustre-MDT0000 [ 3694.003650] Lustre: server umount lustre-MDT0000 complete [ 3713.795536] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3713.800657] LDISKFS-fs (dm-0): recovery complete [ 3713.814151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.216765] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9fd485f [ 3722.224774] Lustre: MGC192.168.203.107@tcp: Connection restored to 0@lo (at 0@lo) [ 3722.229846] Lustre: Skipped 39 previous similar messages [ 3725.842608] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3727.914094] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3727.916748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3731.721508] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3733.007402] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3744.223402] LustreError: 111394:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3744.240616] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3745.312303] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3750.574704] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 14:11:33 (1780855893) [ 3755.171528] LustreError: 10435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3760.607106] LustreError: 10435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3760.613627] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnecting [ 3760.675551] LustreError: 15599:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 3762.378749] LustreError: 9755:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3767.775309] LustreError: 9755:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3767.786076] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnecting [ 3769.486773] LustreError: 6509:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3774.943273] LustreError: 6509:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3774.957379] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnecting [ 3776.557738] LustreError: 63010:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3781.599236] LustreError: 63010:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3782.956321] LustreError: 10435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3788.255155] LustreError: 10435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3788.264231] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnecting [ 3788.271973] Lustre: Skipped 1 previous similar message [ 3796.260221] LustreError: 6507:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3796.265168] LustreError: 6507:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3801.567374] LustreError: 6507:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3801.577728] LustreError: 6507:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3808.223199] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnecting [ 3808.232560] Lustre: Skipped 2 previous similar messages [ 3816.141756] LustreError: 10435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3816.146779] LustreError: 10435:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3821.535155] LustreError: 10435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3821.540056] LustreError: 10435:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3826.904854] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 14:12:50 (1780855970) [ 3828.022278] LustreError: 63010:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3838.306199] Lustre: lustre-MDT0000: Export ffff8e3fb6003800 already connecting from 192.168.203.7@tcp [ 3843.425585] Lustre: lustre-MDT0000: Export ffff8e3fb6003800 already connecting from 192.168.203.7@tcp [ 3848.546806] Lustre: lustre-MDT0000: Export ffff8e3fb6003800 already connecting from 192.168.203.7@tcp [ 3853.666076] Lustre: lustre-MDT0000: Export ffff8e3fb6003800 already connecting from 192.168.203.7@tcp [ 3853.672586] Lustre: Skipped 1 previous similar message [ 3858.788706] Lustre: lustre-MDT0000: Export ffff8e3fb6003800 already connecting from 192.168.203.7@tcp [ 3868.095415] LustreError: 63010:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3868.100835] Lustre: 63010:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8e3facd93800 x1867358887696640/t0(0) o38->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:0/0 lens 520/416 e 0 to 0 dl 1780855992 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3869.029308] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnecting [ 3869.042608] Lustre: Skipped 3 previous similar messages [ 3869.045110] LustreError: 6508:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3893.605505] Lustre: lustre-MDT0000: Export ffff8e3fb6003800 already connecting from 192.168.203.7@tcp [ 3893.608312] Lustre: Skipped 1 previous similar message [ 3909.127633] LustreError: 6508:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3909.138633] Lustre: 6508:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8e3fb875ce00 x1867358887700992/t0(0) o38->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:0/0 lens 520/416 e 0 to 0 dl 1780856033 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3914.084487] LustreError: 9755:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3938.671597] Lustre: lustre-MDT0000: Export ffff8e3fb6003800 already connecting from 192.168.203.7@tcp [ 3938.680218] Lustre: Skipped 3 previous similar messages [ 3954.167985] LustreError: 9755:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3954.174336] Lustre: 9755:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8e3e843e8000 x1867358887704704/t0(0) o38->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:0/0 lens 520/416 e 0 to 0 dl 1780856078 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3959.139772] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnecting [ 3959.150775] Lustre: Skipped 1 previous similar message [ 3959.157512] LustreError: 63010:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3983.715169] Lustre: lustre-MDT0000: Export ffff8e3fb6003800 already connecting from 192.168.203.7@tcp [ 3983.728164] Lustre: Skipped 3 previous similar messages [ 3999.175170] LustreError: 63010:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3999.182474] Lustre: 63010:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8e3fb0e2f480 x1867358887708160/t0(0) o38->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:0/0 lens 520/416 e 0 to 0 dl 1780856123 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4003.495651] LustreError: 6507:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4043.583636] LustreError: 6507:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4043.590240] Lustre: 6507:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8e3fad3c8a80 x1867358887711488/t0(0) o38->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:0/0 lens 520/416 e 0 to 0 dl 1780856168 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4044.645195] LustreError: 6509:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4047.256087] LustreError: 6509:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout interrupted [ 4050.772717] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 14:16:34 (1780856194) [ 4056.605565] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4060.033234] Lustre: Failing over lustre-MDT0000 [ 4060.195763] Lustre: server umount lustre-MDT0000 complete [ 4060.642823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.659330] Lustre: Skipped 24 previous similar messages [ 4070.628555] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4070.631824] LDISKFS-fs (dm-0): recovery complete [ 4070.638955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.739592] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4070.746586] LustreError: Skipped 2 previous similar messages [ 4070.889460] Lustre: *** cfs_fail_loc=712, val=0*** [ 4070.895539] LustreError: 42106:0:(service.c:1397:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8e3fb4e32a00 x1867358905375360/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4070.963378] Lustre: lustre-MDT0000: Aborting client recovery [ 4070.966822] LustreError: 116136:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4070.972840] Lustre: 116169:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4070.981302] Lustre: 116169:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 4070.988407] Lustre: 116169:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4070.996570] Lustre: 116169:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4071.001552] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4071.012274] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4071.041307] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4071.117703] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 4071.121297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 4074.343968] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4076.007571] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4089.011280] Lustre: Failing over lustre-MDT0000 [ 4089.335924] Lustre: server umount lustre-MDT0000 complete [ 4105.925635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.083911] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (not set up) [ 4107.831148] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4107.846469] Lustre: Skipped 3 previous similar messages [ 4109.517772] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4111.343027] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4111.354042] Lustre: Skipped 3 previous similar messages [ 4111.384171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 4111.386077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 4115.630571] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4117.029607] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4123.673705] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 14:17:47 (1780856267) [ 4123.799177] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnecting [ 4123.805557] Lustre: Skipped 2 previous similar messages [ 4130.246813] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 14:17:53 (1780856273) [ 4131.054718] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4131.060532] LustreError: 6517:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3fb4a6b100 x1867358887772416/t0(0) o700->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:36/0 lens 264/248 e 0 to 0 dl 1780856286 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4149.666891] Lustre: Failing over lustre-MDT0000 [ 4149.866429] Lustre: server umount lustre-MDT0000 complete [ 4151.662195] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4151.681185] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 101 previous similar messages [ 4166.738243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.063952] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4167.070615] Lustre: Skipped 6 previous similar messages [ 4169.983871] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4172.305154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 4172.305634] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 4175.716813] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4177.148979] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4186.598778] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 14:18:50 (1780856330) [ 4188.716245] Lustre: Failing over lustre-OST0000 [ 4188.838840] Lustre: server umount lustre-OST0000 complete [ 4205.490775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4209.992730] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4215.586386] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4216.995446] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4287.175205] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 14:20:30 (1780856430) [ 4292.805130] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4294.714289] Lustre: Failing over lustre-MDT0000 [ 4294.868828] Lustre: server umount lustre-MDT0000 complete [ 4295.136836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4295.141366] LustreError: Skipped 7 previous similar messages [ 4314.079106] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780856442/real 1780856442] req@ffff8e3e82c3f100 x1867358905511680/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780856458 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4314.098769] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4314.381387] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4314.385742] LDISKFS-fs (dm-0): recovery complete [ 4314.398679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.300102] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9fd7434 [ 4323.311874] Lustre: MGC192.168.203.107@tcp: Connection restored to 0@lo (at 0@lo) [ 4323.317941] Lustre: Skipped 19 previous similar messages [ 4323.554471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.558659] Lustre: Skipped 5 previous similar messages [ 4326.557448] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4329.264179] Lustre: *** cfs_fail_loc=216, val=0*** [ 4329.267104] LustreError: 121943:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 4329.267477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4330.335583] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4396.349886] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 14:22:20 (1780856540) [ 4397.641559] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4397.650170] Lustre: Skipped 2 previous similar messages [ 4407.864287] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 14:22:31 (1780856551) [ 4410.064793] Lustre: Failing over lustre-MDT0000 [ 4410.224815] Lustre: server umount lustre-MDT0000 complete [ 4426.638557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4430.053646] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4432.364391] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4432.368102] LustreError: 123504:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3fa9668700 x1867358887899264/t0(0) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:376/0 lens 328/344 e 0 to 0 dl 1780856626 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4487.009932] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [ 4487.032658] Lustre: 123504:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4487.085318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 4487.086700] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 4490.806354] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4491.998663] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4499.724773] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 14:24:03 (1780856643) [ 4501.614801] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4507.679581] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4509.279939] Lustre: Failing over lustre-MDT0000 [ 4509.503772] Lustre: server umount lustre-MDT0000 complete [ 4528.713087] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4528.718486] LDISKFS-fs (dm-0): recovery complete [ 4528.725072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.874369] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4534.288173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3137) [ 4534.288380] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 4537.678926] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4538.882418] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4545.236950] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 14:24:48 (1780856688) [ 4546.123907] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4552.757326] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4553.997428] Lustre: Failing over lustre-MDT0000 [ 4554.118969] Lustre: server umount lustre-MDT0000 complete [ 4572.369230] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4572.372437] LDISKFS-fs (dm-0): recovery complete [ 4572.381191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4580.324236] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3fb4b66680 x1867358905661824/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4580.348067] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 4583.724659] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4586.008903] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3137) [ 4586.009937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 4589.532780] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4590.650657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4597.153757] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 14:25:40 (1780856740) [ 4597.917877] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4604.375703] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4606.057961] Lustre: Failing over lustre-MDT0000 [ 4606.290961] Lustre: server umount lustre-MDT0000 complete [ 4624.917938] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4624.919996] LDISKFS-fs (dm-0): recovery complete [ 4624.927891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4628.203483] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4630.532446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3201) [ 4630.532984] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 4637.954767] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 14:26:21 (1780856781) [ 4639.837329] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4639.843116] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4639.850075] LustreError: 6511:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3e8381a300 x1867358887956352/t257698037777(0) o35->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:545/0 lens 392/456 e 0 to 0 dl 1780856795 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4641.750279] Lustre: Failing over lustre-MDT0000 [ 4641.986454] Lustre: server umount lustre-MDT0000 complete [ 4658.339078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4661.401986] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4663.849679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 4663.851803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3201) [ 4663.853347] Lustre: 6510:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3fad22d180 x1867358887956352/t257698037777(0) o35->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:569/0 lens 392/456 e 0 to 0 dl 1780856819 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4667.001580] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4668.189715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4674.613732] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 14:26:58 (1780856818) [ 4675.408702] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4675.414177] LustreError: 6509:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3e832e2a00 x1867358887970048/t261993005072(0) o36->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:625/0 lens 504/448 e 0 to 0 dl 1780856875 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4683.088238] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4684.605189] Lustre: Failing over lustre-MDT0000 [ 4686.784857] Lustre: server umount lustre-MDT0000 complete [ 4689.376061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.387819] Lustre: Skipped 39 previous similar messages [ 4704.961128] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4704.964562] LDISKFS-fs (dm-0): recovery complete [ 4704.973149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.063882] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4705.068790] LustreError: Skipped 8 previous similar messages [ 4708.451201] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4708.706301] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4708.712497] Lustre: Skipped 8 previous similar messages [ 4710.417421] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 4710.417481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3265) [ 4710.420763] Lustre: 6509:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3fb0c6f480 x1867358887970048/t261993005072(0) o36->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:660/0 lens 504/2880 e 0 to 0 dl 1780856910 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4713.548805] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4714.655881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4720.965988] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 14:27:44 (1780856864) [ 4721.639590] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4721.644199] LustreError: 9755:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3fbf8cb800 x1867358887985536/t266287972368(0) o36->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:671/0 lens 504/448 e 0 to 0 dl 1780856921 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4723.200563] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4727.777927] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4728.999352] Lustre: Failing over lustre-MDT0000 [ 4729.123936] Lustre: server umount lustre-MDT0000 complete [ 4746.410787] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4746.413310] LDISKFS-fs (dm-0): recovery complete [ 4746.421124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4756.258338] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4756.273700] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 210 previous similar messages [ 4756.450970] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbc94d16ff9fdadca [ 4759.443919] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4762.099982] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4762.104637] Lustre: Skipped 9 previous similar messages [ 4762.130332] Lustre: 6511:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3f85460850 x1867358887985920/t266287972369(0) o35->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:667/0 lens 392/456 e 0 to 0 dl 1780856917 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4762.134472] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 4762.134514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3297) [ 4762.149034] Lustre: 6511:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4768.949798] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 14:28:32 (1780856912) [ 4769.681650] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4769.685875] Lustre: Skipped 1 previous similar message [ 4769.690035] LustreError: 6508:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3fbc32e680 x1867358888000384/t270582939664(0) o36->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:719/0 lens 504/448 e 0 to 0 dl 1780856969 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4769.706474] LustreError: 6508:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 1 previous similar message [ 4771.228162] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4771.231738] Lustre: Skipped 1 previous similar message [ 4776.077967] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4777.233171] Lustre: Failing over lustre-MDT0000 [ 4777.427408] Lustre: server umount lustre-MDT0000 complete [ 4794.942889] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4794.948850] LDISKFS-fs (dm-0): recovery complete [ 4794.956798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.218510] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4795.230615] Lustre: Skipped 9 previous similar messages [ 4797.501293] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4800.517450] Lustre: 6509:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3fa9739c00 x1867358888000384/t270582939664(0) o36->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:750/0 lens 504/2880 e 0 to 0 dl 1780857000 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4800.520259] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 4800.521308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3329) [ 4806.362971] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 14:29:10 (1780856950) [ 4806.955421] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4808.460292] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4808.462226] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4808.467473] LustreError: 6510:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3f9181b450 x1867358888013696/t274877906960(0) o35->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:714/0 lens 392/456 e 0 to 0 dl 1780856964 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4813.788772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4814.933538] Lustre: Failing over lustre-MDT0000 [ 4815.072738] Lustre: server umount lustre-MDT0000 complete [ 4831.870787] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4831.874423] LDISKFS-fs (dm-0): recovery complete [ 4831.884368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4834.509477] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4837.378779] Lustre: 6511:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3facc6ea00 x1867358888013696/t274877906960(0) o35->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:743/0 lens 392/456 e 0 to 0 dl 1780856993 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4837.382224] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 4837.383671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3361) [ 4843.824161] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 14:29:47 (1780856987) [ 4844.526977] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4844.530418] LustreError: 9755:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3fb0c6d180 x1867358888024320/t279172874255(0) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:39/0 lens 664/608 e 0 to 0 dl 1780857044 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4904.831777] Lustre: 6507:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3e84f70000 x1867358888024320/t279172874255(0) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:99/0 lens 664/3488 e 0 to 0 dl 1780857104 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4909.081600] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 14:30:52 (1780857052) [ 4913.549955] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4914.643520] Lustre: Failing over lustre-MDT0000 [ 4914.767733] Lustre: server umount lustre-MDT0000 complete [ 4931.821116] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4931.823108] LDISKFS-fs (dm-0): recovery complete [ 4931.834218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4932.083319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4932.088831] Lustre: Skipped 9 previous similar messages [ 4934.523509] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4937.185728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4937.190400] Lustre: Skipped 43 previous similar messages [ 4937.272523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 4937.272723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3361) [ 4939.358952] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4940.430661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4956.310733] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 14:31:40 (1780857100) [ 4961.824412] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4963.040105] Lustre: Failing over lustre-MDT0000 [ 4963.265581] Lustre: server umount lustre-MDT0000 complete [ 4967.908772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4967.914953] LustreError: Skipped 8 previous similar messages [ 4981.344362] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4981.346696] LDISKFS-fs (dm-0): recovery complete [ 4981.355467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4984.269561] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4986.924331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3425) [ 4986.924387] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 4989.214629] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4990.141132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4993.427624] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5000.646294] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 14:32:24 (1780857144) [ 5032.019267] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5033.298315] Lustre: Failing over lustre-MDT0000 [ 5033.624207] Lustre: server umount lustre-MDT0000 complete [ 5050.043452] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5050.045469] LDISKFS-fs (dm-0): recovery complete [ 5050.050955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.140976] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5055.657396] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 5055.657814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 5057.599041] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5058.546792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5091.109522] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 14:33:55 (1780857235) [ 5096.091870] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5097.187369] Lustre: Failing over lustre-MDT0000 [ 5097.336329] Lustre: server umount lustre-MDT0000 complete [ 5113.682053] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5113.684019] LDISKFS-fs (dm-0): recovery complete [ 5113.690142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.993941] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5119.010158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 5119.010175] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 5120.910544] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5121.683160] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5125.050359] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5125.842771] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5128.806753] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 14:34:32 (1780857272) [ 5135.056595] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5194.594381] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnecting [ 5194.601861] Lustre: Skipped 2 previous similar messages [ 5195.920755] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5195.923513] Lustre: Skipped 1 previous similar message [ 5195.928412] LustreError: 6508:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3f88b14050 x1867358890817152/t296352743435(0) o36->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:390/0 lens 66040/440 e 0 to 0 dl 1780857395 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5256.042437] Lustre: 63010:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3facfbea00 x1867358890817152/t296352743435(0) o36->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:450/0 lens 66040/440 e 0 to 0 dl 1780857455 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5261.147208] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5261.927332] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 14:36:45 (1780857405) [ 5268.714962] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5270.174197] Lustre: Failing over lustre-MDT0000 [ 5270.386676] Lustre: server umount lustre-MDT0000 complete [ 5286.698664] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5286.701415] LDISKFS-fs (dm-0): recovery complete [ 5286.706844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5289.131545] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5292.509619] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 5292.510099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 5294.224971] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5295.001231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5300.098533] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 14:37:24 (1780857444) [ 5311.478482] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5317.603396] Lustre: Failing over lustre-OST0000 [ 5317.668138] Lustre: server umount lustre-OST0000 complete [ 5318.112403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5318.121544] Lustre: Skipped 36 previous similar messages [ 5333.223287] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5333.225078] LDISKFS-fs (dm-2): recovery complete [ 5333.230744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5333.861280] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5333.865137] Lustre: Skipped 8 previous similar messages [ 5335.791946] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5347.771897] Lustre: Failing over lustre-OST0000 [ 5347.830985] Lustre: server umount lustre-OST0000 complete [ 5359.072305] LustreError: 42102:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5359.084139] LustreError: 42102:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 145 previous similar messages [ 5361.703887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5363.826374] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5363.831894] Lustre: Skipped 8 previous similar messages [ 5364.717537] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5368.843728] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5369.648145] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5404.978500] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 14:39:08 (1780857548) [ 5406.308476] Lustre: Failing over lustre-MDT0000 [ 5406.514795] Lustre: server umount lustre-MDT0000 complete [ 5407.714793] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780857552 with bad export cookie 13588716254549639975 [ 5420.617140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.676678] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5420.682639] LustreError: Skipped 8 previous similar messages [ 5420.815881] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5420.820670] Lustre: Skipped 8 previous similar messages [ 5422.449216] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5426.174606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 5426.174672] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 5434.468784] Lustre: Failing over lustre-MDT0000 [ 5434.621374] Lustre: server umount lustre-MDT0000 complete [ 5448.739149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5451.318327] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5454.341083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 5454.342684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 5456.077938] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5456.762340] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5461.042730] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 14:40:05 (1780857605) [ 5472.752140] Lustre: Failing over lustre-OST0000 [ 5472.802839] Lustre: server umount lustre-OST0000 complete [ 5487.167738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5490.159668] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5494.347127] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5495.126094] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5500.163344] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 14:40:44 (1780857644) [ 5501.321275] Lustre: Failing over lustre-MDT0000 [ 5501.517538] Lustre: server umount lustre-MDT0000 complete [ 5506.402673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.472631] Lustre: *** cfs_fail_loc=605, val=0*** [ 5506.474679] LustreError: 157280:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc078ca30 failed: rc = -95 [ 5506.482663] LustreError: 157280:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 5506.485800] LustreError: 157280:0:(obd_mount.c:250:lustre_start_simple()) MGS setup error -95 [ 5506.490052] LustreError: 157280:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 5506.493950] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5506.499724] LustreError: 157280:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 5506.507333] Lustre: server umount lustre-MDT0000 complete [ 5506.509217] LustreError: 157280:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5510.538978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5512.808733] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5515.785239] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 5515.787768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 5516.948414] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 14:41:00 (1780857660) [ 5520.859643] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5522.598182] Lustre: Failing over lustre-MDT0000 [ 5522.874614] Lustre: server umount lustre-MDT0000 complete [ 5538.677114] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5538.679956] LDISKFS-fs (dm-0): recovery complete [ 5538.684731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5538.834266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.836627] Lustre: Skipped 10 previous similar messages [ 5540.534203] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5543.784430] Lustre: *** cfs_fail_loc=707, val=0*** [ 5543.907077] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5543.909516] Lustre: Skipped 37 previous similar messages [ 5604.206813] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5604.230468] Lustre: 159433:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5604.237790] Lustre: 159433:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 1 previous similar message [ 5604.400381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 5604.400404] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 5606.081636] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5606.872865] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5611.958155] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 14:42:35 (1780857755) [ 5636.947825] LustreError: 6507:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8e3fc0240380 x1867358891699968/t0(0) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:32/0 lens 664/0 e 0 to 0 dl 1780857792 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5636.960497] LustreError: 6507:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 5647.991140] LustreError: 6507:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5647.998485] LustreError: 6511:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8e3fa945f800 x1867358891700992/t0(0) o35->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:43/0 lens 392/0 e 0 to 0 dl 1780857803 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5653.986308] LustreError: 42094:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8e3fb85f0700 x1867358907156864/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:49/0 lens 544/0 e 0 to 0 dl 1780857809 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 5653.996418] LustreError: 42094:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 5660.851607] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 14:43:24 (1780857804) [ 5685.720094] LustreError: 8396:0:(tgt_handler.c:2833:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 5696.751132] LustreError: 8396:0:(tgt_handler.c:2833:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 5700.809512] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 14:44:04 (1780857844) [ 5724.817070] LustreError: 9755:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8e3fb85f3480 x1867358891723520/t0(0) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:120/0 lens 576/0 e 0 to 0 dl 1780857880 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5724.826327] LustreError: 9755:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 5729.927169] LustreError: 9755:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5729.933347] LustreError: 10435:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8e3fad082680 x1867358891724288/t0(0) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:125/0 lens 664/0 e 0 to 0 dl 1780857885 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5740.951219] LustreError: 6508:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5740.962898] LustreError: 6509:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8e3fb8548700 x1867358891745792/t0(0) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:175/0 lens 664/0 e 0 to 0 dl 1780857935 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5740.972343] LustreError: 6509:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 5753.512042] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 14:44:57 (1780857897) [ 5850.385057] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 14:46:34 (1780857994) [ 5874.489215] LustreError: 63010:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8e3fb854b100 x1867358891802240/t0(0) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:270/0 lens 576/0 e 0 to 0 dl 1780858030 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5874.497238] LustreError: 63010:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 5874.500601] LustreError: 63010:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5874.503923] LustreError: 63010:0:(service.c:2562:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 5874.927178] LustreError: 63010:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5890.875546] LustreError: 6509:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5890.882373] LustreError: 6509:0:(service.c:2562:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5891.303142] LustreError: 6509:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5891.306974] LustreError: 6509:0:(service.c:2562:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5906.804752] LustreError: 63010:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8e3fb87f3800 x1867358891821952/t0(0) o36->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:302/0 lens 504/0 e 0 to 0 dl 1780858062 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 5906.818834] LustreError: 63010:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 79 previous similar messages [ 5919.711799] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 14:47:43 (1780858063) [ 5945.340858] Lustre: DEBUG MARKER: phase 2 [ 5948.914656] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 14:48:12 (1780858092) [ 6020.298887] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 14:49:24 (1780858164) [ 6020.804128] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6021.359809] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 14:49:25 (1780858165) [ 6023.161371] Lustre: DEBUG MARKER: Started rundbench load pid=133844 ... [ 6027.059749] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6028.710667] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6029.519375] Lustre: Failing over lustre-MDT0000 [ 6029.741823] Lustre: server umount lustre-MDT0000 complete [ 6030.195945] LustreError: 10435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6030.203573] LustreError: 10435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 65 previous similar messages [ 6030.304248] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6030.310558] Lustre: Skipped 23 previous similar messages [ 6045.331120] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 6045.335761] LDISKFS-fs (dm-0): recovery complete [ 6045.342286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6045.431419] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6045.436457] LustreError: Skipped 3 previous similar messages [ 6045.559550] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6045.566855] Lustre: Skipped 4 previous similar messages [ 6047.548290] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6047.592246] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6047.598623] Lustre: Skipped 6 previous similar messages [ 6050.808783] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6050.814192] Lustre: Skipped 5 previous similar messages [ 6050.831282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 6050.836685] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 6052.538299] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6053.331258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6059.241198] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6061.023076] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6061.879866] Lustre: Failing over lustre-MDT0001 [ 6061.902570] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [ 6062.063402] Lustre: server umount lustre-MDT0001 complete [ 6066.145747] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6066.149741] LustreError: Skipped 10 previous similar messages [ 6076.511739] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 6076.513715] LDISKFS-fs (dm-1): recovery complete [ 6076.519735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.995464] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6082.229927] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:648 to 0x280000400:673) [ 6082.229945] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:631 to 0x2c0000400:673) [ 6083.611668] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6084.265485] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6089.757274] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6091.380657] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 6092.109719] Lustre: Failing over lustre-MDT0000 [ 6092.256719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6092.386763] Lustre: server umount lustre-MDT0000 complete [ 6107.801531] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6107.803761] LDISKFS-fs (dm-0): recovery complete [ 6107.808635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6109.735298] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6113.280081] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 6113.280083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 6114.588883] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6115.305097] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6120.683519] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6122.292256] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 6123.055863] Lustre: Failing over lustre-MDT0001 [ 6123.067033] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [ 6123.070488] Lustre: Skipped 3 previous similar messages [ 6123.080885] LustreError: 162243:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000005a0d67d1 ns: mdt-lustre-MDT0001_UUID lock: ffff8e3e8c949400/0xbc94d16ffa0861f6 lrc: 3/0,0 mode: CW/CW res: [0x2400007ed:0x6ed:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50306400000000 nid: 192.168.203.7@tcp remote: 0x3741e34a663ff52d expref: 4 pid: 162243 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6128.611124] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6128.618761] Lustre: Skipped 6 previous similar messages [ 6129.260777] Lustre: server umount lustre-MDT0001 complete [ 6143.788542] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6143.790267] LDISKFS-fs (dm-1): recovery complete [ 6143.797522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.924879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6143.928141] Lustre: Skipped 3 previous similar messages [ 6145.341261] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6149.090302] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6149.093971] Lustre: Skipped 14 previous similar messages [ 6149.981158] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:916 to 0x280000400:961) [ 6149.981160] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:916 to 0x2c0000400:961) [ 6151.534533] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6152.199359] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6170.251927] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 14:51:54 (1780858314) [ 6293.546056] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6304.402762] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 6305.129488] Lustre: Failing over lustre-MDT0000 [ 6305.411479] Lustre: server umount lustre-MDT0000 complete [ 6320.165929] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6320.167667] LDISKFS-fs (dm-0): recovery complete [ 6320.171893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6321.896596] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6328.716615] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5921 to 0x2c0000401:5953) [ 6328.717657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5953 to 0x280000401:5985) [ 6330.187677] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6330.868074] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6455.321562] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6466.145044] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 6466.851993] Lustre: Failing over lustre-MDT0001 [ 6466.892151] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [ 6467.143958] Lustre: server umount lustre-MDT0001 complete [ 6481.439277] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6481.440494] LDISKFS-fs (dm-1): recovery complete [ 6481.444929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6482.826850] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6488.413616] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2541 to 0x2c0000400:2561) [ 6488.420253] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2541 to 0x280000400:2561) [ 6489.593674] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6490.247537] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6514.371584] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 14:57:38 (1780858658) [ 6637.598844] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6648.303842] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 6648.971167] Lustre: Failing over lustre-MDT0001 [ 6648.982726] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [ 6649.166857] Lustre: server umount lustre-MDT0001 complete [ 6650.848271] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.848648] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6650.853035] Lustre: Skipped 19 previous similar messages [ 6650.858013] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 77 previous similar messages [ 6663.390059] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6663.391392] LDISKFS-fs (dm-1): recovery complete [ 6663.395760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.511862] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6663.513937] Lustre: Skipped 5 previous similar messages [ 6664.783125] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6665.126879] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6665.130512] Lustre: Skipped 5 previous similar messages [ 6669.634307] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6669.636387] Lustre: Skipped 5 previous similar messages [ 6669.649784] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3049 to 0x2c0000400:3073) [ 6669.649823] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3049 to 0x280000400:3073) [ 6670.856051] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6671.417132] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6795.809847] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6806.631879] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 6807.275431] Lustre: Failing over lustre-MDT0000 [ 6807.278116] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -19 [ 6807.281232] LustreError: Skipped 4 previous similar messages [ 6807.287620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6813.269395] Lustre: server umount lustre-MDT0000 complete [ 6827.354828] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6827.357312] LDISKFS-fs (dm-0): recovery complete [ 6827.361044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6827.400575] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6827.403830] LustreError: Skipped 2 previous similar messages [ 6827.471229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6827.474280] Lustre: Skipped 3 previous similar messages [ 6828.779693] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6832.609320] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6832.611090] Lustre: Skipped 12 previous similar messages [ 6833.457628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7587 to 0x280000401:7617) [ 6833.457845] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7555 to 0x2c0000401:7585) [ 6834.644657] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6835.200056] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6838.513476] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 15:03:02 (1780858982) [ 6961.704081] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6972.397050] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 6972.966906] Lustre: Failing over lustre-MDT0000 [ 6973.267857] Lustre: server umount lustre-MDT0000 complete [ 6987.076022] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6987.077439] LDISKFS-fs (dm-0): recovery complete [ 6987.080945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6988.304497] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6995.087746] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9474 to 0x2c0000401:9505) [ 6995.087807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9506 to 0x280000401:9537) [ 6996.374770] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6996.956323] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7121.047323] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7131.722335] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 7132.273460] Lustre: Failing over lustre-MDT0000 [ 7132.302561] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 7132.304636] Lustre: Skipped 5 previous similar messages [ 7132.455884] Lustre: server umount lustre-MDT0000 complete [ 7146.477947] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7146.479132] LDISKFS-fs (dm-0): recovery complete [ 7146.481872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7147.951492] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7154.903531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11621 to 0x280000401:11649) [ 7154.904884] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11588 to 0x2c0000401:11617) [ 7156.145494] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7156.826348] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7160.745233] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 15:08:24 (1780859304) [ 7166.587413] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7168.075522] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7168.718362] Lustre: Failing over lustre-OST0000 [ 7168.762339] Lustre: server umount lustre-OST0000 complete [ 7182.921792] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7182.923677] LDISKFS-fs (dm-2): recovery complete [ 7182.927030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7184.937199] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7188.202799] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7188.815108] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7197.479388] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7198.934804] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 7199.519649] Lustre: Failing over lustre-OST0000 [ 7199.582036] Lustre: server umount lustre-OST0000 complete [ 7213.697365] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7213.699195] LDISKFS-fs (dm-2): recovery complete [ 7213.702755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7215.657213] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7219.094491] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7219.706738] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7225.433479] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 15:09:29 (1780859369) [ 7348.384731] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7350.977716] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7361.736350] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 7362.404459] Lustre: Failing over lustre-MDT0000 [ 7362.463088] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 7362.527786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.531047] Lustre: Skipped 18 previous similar messages [ 7362.633336] Lustre: server umount lustre-MDT0000 complete [ 7363.866603] LustreError: 9471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859508 with bad export cookie 13588716254555503413 [ 7363.867682] Lustre: Failing over lustre-MDT0001 [ 7363.871567] LustreError: 9471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7364.236338] Lustre: server umount lustre-MDT0001 complete [ 7379.012373] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7379.015509] LDISKFS-fs (dm-1): recovery complete [ 7379.019198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7379.029439] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7379.032552] LDISKFS-fs (dm-0): recovery complete [ 7379.037688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7379.099912] LustreError: 189660:0:(llog.c:1646:llog_backup()) MGC192.168.203.107@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7379.102721] Lustre: 189660:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.107@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7381.857094] LustreError: 189667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7381.862829] LustreError: 189667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 71 previous similar messages [ 7384.031138] Lustre: 3655:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780859512/real 1780859512] req@ffff8e3e8da2ca80 x1867358944198016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780859528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7384.041884] Lustre: 3655:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 7389.151447] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3e8bc2d180 x1867358944200960/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7389.231814] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7389.234874] Lustre: Skipped 5 previous similar messages [ 7390.499430] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7390.502244] Lustre: Skipped 6 previous similar messages [ 7390.647356] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7390.736082] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7395.368725] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7395.373766] Lustre: Skipped 5 previous similar messages [ 7395.390719] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3438 to 0x2c0000400:3457) [ 7395.390741] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3437 to 0x280000400:3457) [ 7400.430922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:11745) [ 7400.432175] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11682 to 0x2c0000401:11713) [ 7401.783967] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7402.352327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7402.878282] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7526.986051] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7529.532801] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7540.238835] Lustre: DEBUG MARKER: fail mds1 mds2 2 times [ 7540.868470] Lustre: Failing over lustre-MDT0000 [ 7540.874241] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 7540.878748] Lustre: Skipped 4 previous similar messages [ 7541.064956] Lustre: server umount lustre-MDT0000 complete [ 7542.230575] LustreError: 42104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859687 with bad export cookie 13588716254556534569 [ 7542.231825] Lustre: Failing over lustre-MDT0001 [ 7542.232282] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7542.232288] LustreError: Skipped 3 previous similar messages [ 7542.235355] LustreError: 42104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7542.395427] Lustre: server umount lustre-MDT0001 complete [ 7557.358436] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7557.361793] LDISKFS-fs (dm-0): recovery complete [ 7557.367075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.376261] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7557.379208] LDISKFS-fs (dm-1): recovery complete [ 7557.384186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.671126] Lustre: 3655:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780859689/real 1780859689] req@ffff8e3e8f89c000 x1867358954995072/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780859705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7560.679883] Lustre: 3655:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 7567.839533] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3e8f89c000 x1867358954998016/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7567.944880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.946765] Lustre: Skipped 6 previous similar messages [ 7568.038854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 7568.042754] LustreError: Skipped 11 previous similar messages [ 7569.399618] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7569.562336] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7571.169456] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7571.173362] Lustre: Skipped 21 previous similar messages [ 7572.216311] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3438 to 0x2c0000400:3489) [ 7572.216334] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3437 to 0x280000400:3489) [ 7583.549169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:11777) [ 7583.549175] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11682 to 0x2c0000401:11745) [ 7585.018917] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7585.586674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7586.118711] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7589.460823] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 15:15:33 (1780859733) [ 7591.892908] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7592.771859] Lustre: Failing over lustre-MDT0000 [ 7592.915256] Lustre: server umount lustre-MDT0000 complete [ 7606.839647] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7606.841055] LDISKFS-fs (dm-0): recovery complete [ 7606.844357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7608.221963] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7609.186498] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7624.553984] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7624.567849] Lustre: 192916:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3fadcbd880 x1867358958386048/t343597384332(343597384332) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:510/0 lens 520/3488 e 0 to 0 dl 1780859780 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7624.609310] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11747 to 0x2c0000401:11777) [ 7624.609320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:11809) [ 7625.912826] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7626.458481] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7629.759868] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 15:16:13 (1780859773) [ 7632.194305] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7633.071459] Lustre: Failing over lustre-MDT0000 [ 7633.305776] Lustre: server umount lustre-MDT0000 complete [ 7647.267764] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7647.269026] LDISKFS-fs (dm-0): recovery complete [ 7647.271895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7648.583534] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7650.154993] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7650.156582] LustreError: 192916:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3e843ebb80 x1867358958386048/t343597384332(343597384332) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:535/0 lens 520/664 e 0 to 0 dl 1780859805 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7665.513365] Lustre: lustre-MDT0000: Client 3b8e3256-46d0-40e0-9570-180f14e2d182 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7665.518819] Lustre: 192915:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3fb52fdc00 x1867358958386048/t343597384332(343597384332) o101->3b8e3256-46d0-40e0-9570-180f14e2d182@192.168.203.7@tcp:551/0 lens 520/3488 e 0 to 0 dl 1780859821 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7665.551738] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11779 to 0x2c0000401:11809) [ 7665.551744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:11841) [ 7666.867282] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7667.419110] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7670.741135] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 15:16:54 (1780859814) [ 7672.438211] Lustre: Failing over lustre-OST0000 [ 7672.479112] Lustre: server umount lustre-OST0000 complete [ 7673.722150] Lustre: Failing over lustre-MDT0000 [ 7673.818061] Lustre: server umount lustre-MDT0000 complete [ 7686.683329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.093612] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7688.860894] Lustre: lustre-MDT0000: Denying connection for new client f62bb0d7-0e47-4599-bf2a-18c6d17ddde1 (at 192.168.203.7@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7692.279308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11779 to 0x2c0000401:11841) [ 7696.104121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7697.957466] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7701.476803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:11873) [ 7701.844662] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 15:17:26 (1780859846) [ 7702.198369] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7702.200961] LustreError: 192920:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3e83934a80 x1867358956731648/t365072220167(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:587/0 lens 1312/4320 e 0 to 0 dl 1780859857 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7704.601845] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7705.232406] Lustre: Failing over lustre-MDT0000 [ 7705.646971] Lustre: server umount lustre-MDT0000 complete [ 7717.343086] Lustre: 192969:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780859846/real 1780859846] req@ffff8e3e83935880 x1867358956731648/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1780859862 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7717.355023] Lustre: 192969:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 7719.605037] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7719.606277] LDISKFS-fs (dm-0): recovery complete [ 7719.609333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7720.824864] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7725.047261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:11905) [ 7725.047270] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:11873) [ 7726.199352] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7726.734733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7730.308370] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 15:17:54 (1780859874) [ 7733.182409] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7735.691930] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7736.291724] Lustre: Failing over lustre-MDT0001 [ 7742.021895] Lustre: server umount lustre-MDT0001 complete [ 7756.101833] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7756.103722] LDISKFS-fs (dm-1): recovery complete [ 7756.107614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7757.506982] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7761.392971] Lustre: 193654:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3facf8aa00 x1867358972987904/t30064773788(0) o36->f62bb0d7-0e47-4599-bf2a-18c6d17ddde1@192.168.203.7@tcp:669/0 lens 560/2880 e 0 to 0 dl 1780859939 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7761.394717] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3500 to 0x280000400:3521) [ 7761.394776] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3500 to 0x2c0000400:3521) [ 7762.541397] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7763.065500] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7766.552161] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 15:18:30 (1780859910) [ 7766.881754] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7766.884264] Lustre: Skipped 1 previous similar message [ 7766.886496] LustreError: 193659:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3fc0241180 x1867358956796032/t369367187489(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:652/0 lens 2520/4320 e 0 to 0 dl 1780859922 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7766.895800] LustreError: 193659:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 1 previous similar message [ 7769.232750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7771.621452] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7772.247890] Lustre: Failing over lustre-MDT0000 [ 7772.624180] Lustre: server umount lustre-MDT0000 complete [ 7786.533754] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7786.534968] LDISKFS-fs (dm-0): recovery complete [ 7786.537316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7787.872616] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7792.143407] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:11905) [ 7792.143424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:11937) [ 7793.312958] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7793.850720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7795.752253] Lustre: Failing over lustre-MDT0001 [ 7795.838917] Lustre: server umount lustre-MDT0001 complete [ 7809.764955] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7809.767066] LDISKFS-fs (dm-1): recovery complete [ 7809.771418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.126762] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7815.153694] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3532 to 0x2c0000400:3553) [ 7815.153726] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3532 to 0x280000400:3553) [ 7816.247043] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7816.782431] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7820.263208] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 15:19:24 (1780859964) [ 7825.998670] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7828.471921] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7829.103922] Lustre: Failing over lustre-MDT0000 [ 7829.350231] Lustre: server umount lustre-MDT0000 complete [ 7830.566689] LustreError: 6492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780859975 with bad export cookie 13588716254557537921 [ 7830.567628] Lustre: Failing over lustre-MDT0001 [ 7830.569876] LustreError: 6492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7830.575368] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [ 7830.577208] Lustre: Skipped 5 previous similar messages [ 7836.127749] LustreError: 208585:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8e3fa94a8380 x1867358956858752/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7836.134084] LustreError: 208585:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7836.230333] Lustre: server umount lustre-MDT0001 complete [ 7850.878665] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7850.881619] LDISKFS-fs (dm-0): recovery complete [ 7850.882518] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7850.884753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.885528] LDISKFS-fs (dm-1): recovery complete [ 7850.893878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.968464] LustreError: 209686:0:(llog.c:1646:llog_backup()) MGC192.168.203.107@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7850.972667] Lustre: 209686:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.107@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7856.095434] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3e8fbb3100 x1867358956859904/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7857.608960] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7857.711979] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7861.303496] Lustre: 209695:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3fadab8e00 x1867358973055104/t38654705732(0) o36->f62bb0d7-0e47-4599-bf2a-18c6d17ddde1@192.168.203.7@tcp:16/0 lens 560/2880 e 0 to 0 dl 1780860041 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7861.309225] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3564 to 0x280000400:3585) [ 7861.311368] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3564 to 0x2c0000400:3585) [ 7861.346852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:11969) [ 7861.346870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:11937) [ 7862.558807] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7863.119579] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7863.674436] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7867.217985] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 15:20:11 (1780860011) [ 7867.569587] LustreError: 209696:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3e86bf5c00 x1867358973088384/t42949673028(0) o36->f62bb0d7-0e47-4599-bf2a-18c6d17ddde1@192.168.203.7@tcp:753/0 lens 560/448 e 0 to 0 dl 1780860023 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7867.577051] LustreError: 209696:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 1 previous similar message [ 7872.989596] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7873.615908] Lustre: Failing over lustre-MDT0000 [ 7873.977061] Lustre: server umount lustre-MDT0000 complete [ 7884.145245] Lustre: lustre-MDT0001: Client f62bb0d7-0e47-4599-bf2a-18c6d17ddde1 (at 192.168.203.7@tcp) reconnecting [ 7884.148458] Lustre: Skipped 1 previous similar message [ 7887.996967] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7887.998515] LDISKFS-fs (dm-0): recovery complete [ 7888.001277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7889.464614] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7893.501491] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:11969) [ 7893.501491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:12001) [ 7894.685194] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7895.202529] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7898.566658] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 15:20:42 (1780860042) [ 7898.886593] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7898.888424] Lustre: Skipped 2 previous similar messages [ 7901.098641] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7901.654685] Lustre: Failing over lustre-MDT0001 [ 7901.728597] Lustre: server umount lustre-MDT0001 complete [ 7915.604459] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7915.606377] LDISKFS-fs (dm-1): recovery complete [ 7915.612128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.998932] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7921.139136] Lustre: 209696:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3e8da5bb80 x1867358973118464/t42949673097(0) o36->f62bb0d7-0e47-4599-bf2a-18c6d17ddde1@192.168.203.7@tcp:51/0 lens 560/2880 e 0 to 0 dl 1780860076 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7921.144807] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3606 to 0x280000400:3649) [ 7921.144973] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3606 to 0x2c0000400:3649) [ 7921.156971] Lustre: 209696:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7922.231580] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7922.813812] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7926.375347] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 15:21:10 (1780860070) [ 7932.106678] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7934.594619] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7935.200939] Lustre: Failing over lustre-MDT0000 [ 7935.454539] Lustre: server umount lustre-MDT0000 complete [ 7949.541212] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7949.543148] LDISKFS-fs (dm-0): recovery complete [ 7949.546465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7950.963968] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7954.942898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:12033) [ 7954.942898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:12001) [ 7956.077726] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7956.667828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7958.678286] Lustre: Failing over lustre-MDT0001 [ 7958.777393] Lustre: server umount lustre-MDT0001 complete [ 7972.780812] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7972.781928] LDISKFS-fs (dm-1): recovery complete [ 7972.786586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7974.126388] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7977.977405] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3660 to 0x280000400:3681) [ 7977.977405] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3660 to 0x2c0000400:3681) [ 7979.081933] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7979.621069] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7983.091137] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 15:22:07 (1780860127) [ 7988.821414] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7991.284381] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7991.896888] Lustre: Failing over lustre-MDT0000 [ 7992.160705] Lustre: server umount lustre-MDT0000 complete [ 7993.311761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7993.312207] LustreError: 209696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7993.315824] Lustre: Skipped 56 previous similar messages [ 7993.319879] LustreError: 209696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 177 previous similar messages [ 7993.378802] LustreError: 9471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860138 with bad export cookie 13588716254557551613 [ 7993.379565] Lustre: Failing over lustre-MDT0001 [ 7993.383696] LustreError: 9471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7993.476535] Lustre: server umount lustre-MDT0001 complete [ 8008.101922] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8008.102049] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8008.103703] LDISKFS-fs (dm-0): recovery complete [ 8008.105072] LDISKFS-fs (dm-1): recovery complete [ 8008.108540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8008.111247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8008.191741] LustreError: 220249:0:(llog.c:1646:llog_backup()) MGC192.168.203.107@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8008.196508] Lustre: 220249:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.107@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8018.991554] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8018.995134] Lustre: Skipped 17 previous similar messages [ 8020.127115] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780860148/real 1780860148] req@ffff8e3e8bc30a80 x1867358957015168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780860164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8020.136529] Lustre: 3652:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 8020.321460] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8020.324397] Lustre: Skipped 17 previous similar messages [ 8020.530696] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8020.674424] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8025.126297] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8025.130079] Lustre: Skipped 17 previous similar messages [ 8025.134754] Lustre: 221005:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3fb4bbd500 x1867358973183744/t51539607620(0) o36->f62bb0d7-0e47-4599-bf2a-18c6d17ddde1@192.168.203.7@tcp:155/0 lens 560/2880 e 0 to 0 dl 1780860180 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8025.145193] Lustre: 221005:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8025.145359] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3692 to 0x280000400:3713) [ 8025.145459] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3692 to 0x2c0000400:3713) [ 8025.171369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:12033) [ 8025.171376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:12065) [ 8026.433189] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8027.008108] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8027.544754] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8031.247874] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 15:22:55 (1780860175) [ 8031.633694] LustreError: 172698:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3e86bf6680 x1867358957047040/t390842023950(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:162/0 lens 1728/4320 e 0 to 0 dl 1780860187 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8031.643023] LustreError: 172698:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 3 previous similar messages [ 8034.093928] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8034.726270] Lustre: Failing over lustre-MDT0001 [ 8041.005804] Lustre: server umount lustre-MDT0001 complete [ 8055.179176] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8055.180477] LDISKFS-fs (dm-1): recovery complete [ 8055.184337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.592851] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8060.400860] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3724 to 0x280000400:3745) [ 8060.400931] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3724 to 0x2c0000400:3745) [ 8061.556956] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8062.098318] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8065.515052] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 15:23:29 (1780860209) [ 8068.376248] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8068.996866] Lustre: Failing over lustre-MDT0000 [ 8069.344333] Lustre: server umount lustre-MDT0000 complete [ 8083.341595] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8083.342873] LDISKFS-fs (dm-0): recovery complete [ 8083.346459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8084.701620] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8088.574293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:12097) [ 8088.574308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:12065) [ 8089.704500] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8090.214773] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8093.643521] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 15:23:57 (1780860237) [ 8096.438841] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8098.980634] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8099.597940] Lustre: Failing over lustre-MDT0000 [ 8099.961414] Lustre: server umount lustre-MDT0000 complete [ 8113.902647] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8113.903860] LDISKFS-fs (dm-0): recovery complete [ 8113.907497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8115.292869] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8119.298710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:12129) [ 8119.298709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:12097) [ 8120.390737] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8120.906155] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8122.818410] Lustre: Failing over lustre-MDT0001 [ 8122.907501] Lustre: server umount lustre-MDT0001 complete [ 8136.865355] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8136.866716] LDISKFS-fs (dm-1): recovery complete [ 8136.871572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8138.201061] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8142.322735] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3724 to 0x2c0000400:3777) [ 8142.322741] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3724 to 0x280000400:3777) [ 8143.434917] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8143.990777] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8147.345330] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 15:24:51 (1780860291) [ 8150.098570] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8152.551464] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8153.153340] Lustre: Failing over lustre-MDT0000 [ 8153.413368] Lustre: server umount lustre-MDT0000 complete [ 8154.602648] LustreError: 42101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860299 with bad export cookie 13588716254557562036 [ 8154.603740] Lustre: Failing over lustre-MDT0001 [ 8154.604487] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8154.604492] LustreError: Skipped 11 previous similar messages [ 8154.608821] LustreError: 42101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8159.711986] LustreError: 229716:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8e3e86d40a80 x1867358957159680/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8159.719225] LustreError: 229716:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8159.820117] Lustre: server umount lustre-MDT0001 complete [ 8174.647342] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8174.648654] LDISKFS-fs (dm-0): recovery complete [ 8174.651503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8174.720301] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8174.721816] LDISKFS-fs (dm-1): recovery complete [ 8174.725888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8180.191375] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8e3e8e2be680 x1867358957160448/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8180.197289] LustreError: 3651:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 8180.306219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8180.308079] Lustre: Skipped 21 previous similar messages [ 8180.384764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 8180.388226] LustreError: Skipped 16 previous similar messages [ 8181.715414] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8181.816807] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8184.544727] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 8184.546721] Lustre: Skipped 72 previous similar messages [ 8185.482669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:12161) [ 8185.482669] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:12129) [ 8185.496398] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3724 to 0x280000400:3809) [ 8185.496540] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3724 to 0x2c0000400:3809) [ 8185.509196] Lustre: 230832:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e3fa94b5880 x1867358973259904/t64424509442(0) o36->f62bb0d7-0e47-4599-bf2a-18c6d17ddde1@192.168.203.7@tcp:341/0 lens 496/2888 e 0 to 0 dl 1780860366 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8185.514875] Lustre: 230832:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8186.668464] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8187.200980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8187.695345] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8191.071255] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 15:25:35 (1780860335) [ 8191.433320] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8191.434598] Lustre: Skipped 6 previous similar messages [ 8194.159093] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8194.755896] Lustre: Failing over lustre-MDT0000 [ 8194.980391] Lustre: server umount lustre-MDT0000 complete [ 8207.713552] Lustre: lustre-MDT0001: Client f62bb0d7-0e47-4599-bf2a-18c6d17ddde1 (at 192.168.203.7@tcp) reconnecting [ 8207.717055] Lustre: Skipped 1 previous similar message [ 8208.924538] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8208.925788] LDISKFS-fs (dm-0): recovery complete [ 8208.928308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.302552] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8214.525175] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:12161) [ 8214.525175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:12193) [ 8215.689875] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8216.229855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8219.719819] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 15:26:03 (1780860363) [ 8222.534852] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8223.168416] Lustre: Failing over lustre-MDT0001 [ 8223.246203] Lustre: server umount lustre-MDT0001 complete [ 8237.180214] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8237.182552] LDISKFS-fs (dm-1): recovery complete [ 8237.186789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8238.563739] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8242.679318] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3724 to 0x2c0000400:3841) [ 8242.679400] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3724 to 0x280000400:3841) [ 8243.854575] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8244.400207] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8247.800567] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 15:26:31 (1780860391) [ 8250.518633] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8252.952139] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8253.563948] Lustre: Failing over lustre-MDT0000 [ 8253.797731] Lustre: server umount lustre-MDT0000 complete [ 8267.787855] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8267.789964] LDISKFS-fs (dm-0): recovery complete [ 8267.793399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.175732] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8273.411183] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:12193) [ 8273.411266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:12225) [ 8274.566493] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8275.103287] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8276.989777] Lustre: Failing over lustre-MDT0001 [ 8277.069604] Lustre: server umount lustre-MDT0001 complete [ 8291.104087] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8291.105411] LDISKFS-fs (dm-1): recovery complete [ 8291.108990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8292.450242] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8296.434571] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3724 to 0x2c0000400:3873) [ 8296.434572] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3724 to 0x280000400:3873) [ 8297.575291] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8298.095646] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8301.510295] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 15:27:25 (1780860445) [ 8301.881248] LustreError: 230833:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8e3e85c54a80 x1867358973313024/t77309411330(0) o36->f62bb0d7-0e47-4599-bf2a-18c6d17ddde1@192.168.203.7@tcp:432/0 lens 496/456 e 0 to 0 dl 1780860457 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8301.889796] LustreError: 230833:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 6 previous similar messages [ 8304.317461] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8306.772611] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8307.385926] Lustre: Failing over lustre-MDT0000 [ 8307.542299] Lustre: server umount lustre-MDT0000 complete [ 8308.748723] LustreError: 42101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860453 with bad export cookie 13588716254557569596 [ 8308.749957] Lustre: Failing over lustre-MDT0001 [ 8308.754219] LustreError: 42101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8308.851873] Lustre: server umount lustre-MDT0001 complete [ 8323.484944] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8323.487569] LDISKFS-fs (dm-0): recovery complete [ 8323.489249] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8323.491259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8323.492065] LDISKFS-fs (dm-1): recovery complete [ 8323.499940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8334.842964] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8334.967805] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8342.385622] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3724 to 0x280000400:3905) [ 8342.385631] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3724 to 0x2c0000400:3905) [ 8347.518717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11714 to 0x280000401:12257) [ 8347.518722] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11843 to 0x2c0000401:12225) [ 8348.980787] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8349.583974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8350.142305] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8353.665517] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 15:28:17 (1780860497) [ 8354.313550] Lustre: 242675:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting f62bb0d7-0e47-4599-bf2a-18c6d17ddde1 at adminstrative request [ 8358.945566] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 15:28:23 (1780860503) [ 8360.826151] Lustre: Failing over lustre-MDT0000 [ 8360.980652] Lustre: server umount lustre-MDT0000 complete [ 8373.686863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.060727] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8378.892228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12308 to 0x280000401:12353) [ 8378.892246] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12277 to 0x2c0000401:12321) [ 8379.998467] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8380.530060] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8383.870358] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 15:28:48 (1780860528) [ 8387.559723] Lustre: Failing over lustre-OST0000 [ 8387.626579] Lustre: server umount lustre-OST0000 complete [ 8400.325967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8402.160454] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8405.156169] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8405.655450] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8409.114048] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 15:29:13 (1780860553) [ 8410.376879] Lustre: Failing over lustre-MDT0000 [ 8410.592145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8410.594118] Lustre: Skipped 24 previous similar messages [ 8410.631219] Lustre: server umount lustre-MDT0000 complete [ 8413.037903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.335854] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8415.040921] Lustre: lustre-MDT0000: Denying connection for new client 4b455254-560a-4309-88a8-a6def0acc8d8 (at 192.168.203.7@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 8415.046010] Lustre: Skipped 2 previous similar messages [ 8418.293780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12308 to 0x280000401:12385) [ 8418.293789] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12277 to 0x2c0000401:12353) [ 8422.274441] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 15:29:26 (1780860566) [ 8424.954707] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8425.809685] Lustre: Failing over lustre-OST0000 [ 8425.842815] Lustre: server umount lustre-OST0000 complete [ 8439.715791] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8439.717108] LDISKFS-fs (dm-2): recovery complete [ 8439.720031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8441.630469] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8444.657452] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8445.184359] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8448.719658] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 15:29:52 (1780860592) [ 8451.430312] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8453.259121] Lustre: Failing over lustre-OST0000 [ 8453.290316] Lustre: server umount lustre-OST0000 complete [ 8467.139587] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8467.140946] LDISKFS-fs (dm-2): recovery complete [ 8467.144059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8468.973643] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8469.102855] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.7@tcp inode [0x200030971:0x5:0x0] object 0x280000401:12387 extent [0-4194303]: client csum 3abbe2, server csum 28a0f31d [ 8471.968297] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8472.500871] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8475.869803] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 15:30:20 (1780860620) [ 8478.311741] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8480.738640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8482.698364] Lustre: Failing over lustre-MDT0000 [ 8482.898340] Lustre: server umount lustre-MDT0000 complete [ 8484.056784] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780860628 with bad export cookie 13588716254557596399 [ 8484.058478] Lustre: Failing over lustre-OST0000 [ 8484.060641] LustreError: 6493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8486.136460] Lustre: server umount lustre-OST0000 complete [ 8500.222420] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8500.223613] LDISKFS-fs (dm-0): recovery complete [ 8500.226222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.731717] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8513.621811] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12277 to 0x2c0000401:12385) [ 8523.997792] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8523.999177] LDISKFS-fs (dm-2): recovery complete [ 8524.002020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8525.795347] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8532.957711] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 15:31:17 (1780860677) [ 8537.693730] Lustre: Failing over lustre-OST0000 [ 8537.744030] Lustre: server umount lustre-OST0000 complete [ 8538.998239] Lustre: Failing over lustre-MDT0000 [ 8539.375341] Lustre: server umount lustre-MDT0000 complete [ 8552.092321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8553.383424] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8557.562372] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12277 to 0x2c0000401:12417) [ 8563.048616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8564.938476] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8565.641224] Lustre: lustre-OST0000: Denying connection for new client d38804af-5464-4e33-8bfd-ce96f90b0362 (at 192.168.203.7@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 8570.721176] Lustre: lustre-OST0000: Denying connection for new client d38804af-5464-4e33-8bfd-ce96f90b0362 (at 192.168.203.7@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 8580.961318] Lustre: lustre-OST0000: Denying connection for new client d38804af-5464-4e33-8bfd-ce96f90b0362 (at 192.168.203.7@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 8580.966334] Lustre: Skipped 1 previous similar message [ 8601.441207] Lustre: lustre-OST0000: Denying connection for new client d38804af-5464-4e33-8bfd-ce96f90b0362 (at 192.168.203.7@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:33 [ 8601.445714] Lustre: Skipped 3 previous similar messages [ 8634.500218] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 8634.503332] Lustre: 255670:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 4b455254-560a-4309-88a8-a6def0acc8d8@ [ 8634.509032] Lustre: 255670:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 8634.512942] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 8634.519073] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 8634.521794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12436 to 0x280000401:12457) [ 8634.522507] Lustre: Skipped 21 previous similar messages [ 8638.159900] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 8647.304888] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 8649.320182] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 15:33:13 (1780860793) [ 8650.751568] Lustre: Failing over lustre-OST0001 [ 8650.803874] Lustre: server umount lustre-OST0001 complete [ 8652.648526] LustreError: 42096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8652.654616] LustreError: 42096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 235 previous similar messages [ 8654.815558] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8654.819194] Lustre: Skipped 62 previous similar messages [ 8663.927178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8663.980616] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [ 8663.984323] Lustre: Skipped 22 previous similar messages [ 8665.699335] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8665.701485] Lustre: Skipped 21 previous similar messages [ 8665.799723] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8669.785679] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 15:33:33 (1780860813) [ 8671.044852] Lustre: Failing over lustre-OST0000 [ 8671.094178] Lustre: server umount lustre-OST0000 complete [ 8683.800441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8685.539135] LustreError: 258881:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 8685.543034] LustreError: 258881:0:(ldlm_lib.c:2886:target_recovery_thread()) Skipped 77 previous similar messages [ 8685.687287] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8691.679129] Lustre: *** cfs_fail_loc=715, val=40*** [ 8701.729107] Lustre: lustre-OST0000: Client d38804af-5464-4e33-8bfd-ce96f90b0362 (at 192.168.203.7@tcp) reconnected, waiting for 3 clients in recovery for 1:24 [ 8701.919128] Lustre: 3651:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780860830/real 1780860830] req@ffff8e3fbf979880 x1867358957562624/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780860846 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 8701.926732] Lustre: 3651:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 17 previous similar messages [ 8708.063093] Lustre: *** cfs_fail_loc=715, val=40*** [ 8708.064531] Lustre: Skipped 2 previous similar messages [ 8717.279561] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:09 [ 8717.283876] Lustre: Skipped 3 previous similar messages [ 8723.423127] Lustre: *** cfs_fail_loc=715, val=40*** [ 8723.425898] Lustre: Skipped 2 previous similar messages [ 8725.583118] LustreError: 258881:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8725.585871] LustreError: 258881:0:(ldlm_lib.c:2886:target_recovery_thread()) Skipped 77 previous similar messages [ 8726.895940] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8727.420529] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8730.827125] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 15:34:35 (1780860875) [ 8732.116177] Lustre: Failing over lustre-MDT0000 [ 8732.320862] Lustre: server umount lustre-MDT0000 complete [ 8745.049396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8746.445593] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8750.570164] LustreError: 260319:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 8755.167147] Lustre: *** cfs_fail_loc=715, val=80*** [ 8755.169820] Lustre: Skipped 2 previous similar messages [ 8765.281643] Lustre: lustre-MDT0000: Client d38804af-5464-4e33-8bfd-ce96f90b0362 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 1:26 [ 8765.290291] Lustre: Skipped 1 previous similar message [ 8766.943738] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8771.551113] Lustre: *** cfs_fail_loc=715, val=80*** [ 8771.553182] Lustre: Skipped 1 previous similar message [ 8780.641158] Lustre: lustre-MDT0000: Client d38804af-5464-4e33-8bfd-ce96f90b0362 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 1:10 [ 8782.303422] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8786.911148] Lustre: *** cfs_fail_loc=715, val=80*** [ 8786.912602] Lustre: Skipped 1 previous similar message [ 8797.025064] Lustre: lustre-MDT0000: Client d38804af-5464-4e33-8bfd-ce96f90b0362 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 8798.687463] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8803.295118] Lustre: *** cfs_fail_loc=715, val=80*** [ 8803.296334] Lustre: Skipped 1 previous similar message [ 8812.385172] Lustre: lustre-MDT0000: Client d38804af-5464-4e33-8bfd-ce96f90b0362 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 0:39 [ 8815.071539] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8828.768994] Lustre: lustre-MDT0000: Client d38804af-5464-4e33-8bfd-ce96f90b0362 (at 192.168.203.7@tcp) reconnected, waiting for 2 clients in recovery for 0:22 [ 8830.431979] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8830.655118] LustreError: 260319:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8830.660892] Lustre: 260319:0:(ldlm_lib.c:2932:target_recovery_thread()) too long recovery - read logs [ 8830.660966] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8830.663190] LustreError: dumping log to /tmp/lustre-log.1780860975.260319 [ 8830.664623] Lustre: Skipped 56 previous similar messages [ 8830.719139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12430 to 0x2c0000401:12449) [ 8830.719166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12471 to 0x280000401:12489) [ 8831.958526] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8832.480708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8836.085362] Lustre: DEBUG MARKER: == replay-single test complete, duration 8561 sec ======== 15:36:20 (1780860980) [ 8836.591527] Lustre: DEBUG MARKER: === replay-single: start cleanup 15:36:20 (1780860980) === [ 8839.002303] Lustre: DEBUG MARKER: === replay-single: finish cleanup 15:36:23 (1780860983) === [ 8839.665903] Lustre: Failing over lustre-MDT0000 [ 8845.889091] Lustre: server umount lustre-MDT0000 complete [ 8861.246869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.282752] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8861.287722] LustreError: Skipped 8 previous similar messages [ 8861.387907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8861.391251] Lustre: Skipped 19 previous similar messages [ 8862.667482] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8866.816139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12471 to 0x280000401:12521) [ 8866.816157] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12430 to 0x2c0000401:12481) [ 8867.943235] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8868.460336] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8871.904378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8871.907044] LustreError: Skipped 13 previous similar messages [ 8876.883603] Lustre: server umount lustre-MDT0000 complete [ 8879.586629] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780861024 with bad export cookie 13588716254557629236 [ 8879.589992] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8879.692893] Lustre: server umount lustre-MDT0001 complete [ 8892.600356] Lustre: server umount lustre-OST0000 complete [ 8904.474289] Lustre: server umount lustre-OST0001 complete [ 8909.676395] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 8910.608086] Key type lgssc unregistered [ 8910.732329] LNet: 264893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8910.734437] LNetError: 264893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8910.741295] LNet: Removed LNI 192.168.203.107@tcp [ 8911.042122] Key type .llcrypt unregistered [ 8911.043981] Key type ._llcrypt unregistered