[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 546222111 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002639] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007749] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010202] LSM: Security Framework initializing [ 0.011042] Yama: becoming mindful. [ 0.012104] SELinux: Initializing. [ 0.013113] *** VALIDATE selinux *** [ 0.023516] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028258] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029217] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032030] *** VALIDATE tmpfs *** [ 0.033568] *** VALIDATE proc *** [ 0.034214] *** VALIDATE cgroup *** [ 0.035006] *** VALIDATE cgroup2 *** [ 0.037188] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038214] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040025] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.044065] debug: unmapping init [mem 0xffffffffb4059000-0xffffffffb4060fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048802] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049021] ... version: 2 [ 0.050013] ... bit width: 48 [ 0.051009] ... generic registers: 4 [ 0.052012] ... value mask: 0000ffffffffffff [ 0.053012] ... max period: 00007fffffffffff [ 0.054011] ... fixed-purpose events: 3 [ 0.055010] ... event mask: 000000070000000f [ 0.056347] rcu: Hierarchical SRCU implementation. [ 0.059141] smp: Bringing up secondary CPUs ... [ 0.060678] x86: Booting SMP configuration: [ 0.061059] .... node #0, CPUs: #1 #2 #3 [ 0.074228] smp: Brought up 1 node, 4 CPUs [ 0.076012] smpboot: Max logical packages: 1 [ 0.077016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.155397] node 0 deferred pages initialised in 76ms [ 0.162143] devtmpfs: initialized [ 0.163854] x86/mm: Memory block size: 128MB [ 0.166178] gcov: version magic: 0x41383552 [ 0.168185] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.169073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.170247] pinctrl core: initialized pinctrl subsystem [ 0.171188] [ 0.171908] ************************************************************* [ 0.172015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173013] ** ** [ 0.174015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175010] ** ** [ 0.176011] ** This means that this kernel is built to expose internal ** [ 0.177011] ** IOMMU data structures, which may compromise security on ** [ 0.178011] ** your system. ** [ 0.179013] ** ** [ 0.180010] ** If you see this message and you are not debugging the ** [ 0.181011] ** kernel, report this immediately to your vendor! ** [ 0.182015] ** ** [ 0.183013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184009] ************************************************************* [ 0.185943] NET: Registered protocol family 16 [ 0.188426] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.194060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198099] cpuidle: using governor menu [ 0.200604] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.203812] PCI: Using configuration type 1 for base access [ 0.207119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.223503] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225502] cryptd: max_cpu_qlen set to 1000 [ 0.226000] ACPI: Added _OSI(Module Device) [ 0.226000] ACPI: Added _OSI(Processor Device) [ 0.226000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226040] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.234218] ACPI: Interpreter enabled [ 0.235094] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237010] ACPI: Using IOAPIC for interrupt routing [ 0.239099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.242334] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254726] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.258055] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266612] acpiphp: Slot [2] registered [ 0.269278] acpiphp: Slot [5] registered [ 0.270247] acpiphp: Slot [6] registered [ 0.272112] acpiphp: Slot [7] registered [ 0.274099] acpiphp: Slot [8] registered [ 0.275093] acpiphp: Slot [9] registered [ 0.277124] acpiphp: Slot [10] registered [ 0.278092] acpiphp: Slot [3] registered [ 0.280073] acpiphp: Slot [4] registered [ 0.281083] acpiphp: Slot [11] registered [ 0.283096] acpiphp: Slot [12] registered [ 0.285085] acpiphp: Slot [13] registered [ 0.286248] acpiphp: Slot [14] registered [ 0.287074] acpiphp: Slot [15] registered [ 0.289068] acpiphp: Slot [16] registered [ 0.290172] acpiphp: Slot [17] registered [ 0.291075] acpiphp: Slot [18] registered [ 0.293108] acpiphp: Slot [19] registered [ 0.294157] acpiphp: Slot [20] registered [ 0.296077] acpiphp: Slot [21] registered [ 0.297077] acpiphp: Slot [22] registered [ 0.298079] acpiphp: Slot [23] registered [ 0.300074] acpiphp: Slot [24] registered [ 0.301126] acpiphp: Slot [25] registered [ 0.303065] acpiphp: Slot [26] registered [ 0.304073] acpiphp: Slot [27] registered [ 0.305082] acpiphp: Slot [28] registered [ 0.307072] acpiphp: Slot [29] registered [ 0.308086] acpiphp: Slot [30] registered [ 0.310096] acpiphp: Slot [31] registered [ 0.311051] PCI host bridge to bus 0000:00 [ 0.312015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.314017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.317018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.319018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.324021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.325161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.329260] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.332408] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.342013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.349215] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.351014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.354012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.356012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.359571] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.362970] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.365042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.368907] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.374012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.387129] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.393007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.399205] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.414018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.468089] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.478146] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.483012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.488013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.522016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.542450] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.551027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.555011] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.569015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.582469] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.587012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.592014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.605015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.619220] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.629013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.633011] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.645017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.655596] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.662013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.668013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.693113] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.701000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.703275] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.705266] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.707332] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.709200] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.713041] iommu: Default domain type: Passthrough [ 0.714398] SCSI subsystem initialized [ 0.716123] ACPI: bus type USB registered [ 0.717104] usbcore: registered new interface driver usbfs [ 0.719097] usbcore: registered new interface driver hub [ 0.721150] usbcore: registered new device driver usb [ 0.723182] pps_core: LinuxPPS API ver. 1 registered [ 0.725009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.728052] PTP clock support registered [ 0.731040] EDAC MC: Ver: 3.0.0 [ 0.732308] PCI: Using ACPI for IRQ routing [ 0.733391] NetLabel: Initializing [ 0.734006] NetLabel: domain hash size = 128 [ 0.735008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.736054] NetLabel: unlabeled traffic allowed by default [ 0.738228] vgaarb: loaded [ 0.740779] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.752000] clocksource: Switched to clocksource kvm-clock [ 0.843877] VFS: Disk quotas dquot_6.6.0 [ 0.844840] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.846499] *** VALIDATE ramfs *** [ 0.847349] *** VALIDATE hugetlbfs *** [ 0.848193] pnp: PnP ACPI init [ 0.849722] pnp: PnP ACPI: found 6 devices [ 0.865644] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.869279] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.871693] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.873877] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.876618] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.879194] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.881916] NET: Registered protocol family 2 [ 0.885160] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.891219] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.895339] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.900988] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.904587] TCP: Hash tables configured (established 65536 bind 65536) [ 0.907452] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.910684] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.913468] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.916388] NET: Registered protocol family 1 [ 0.919319] RPC: Registered named UNIX socket transport module. [ 0.921504] RPC: Registered udp transport module. [ 0.923204] RPC: Registered tcp transport module. [ 0.924969] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.927600] NET: Registered protocol family 44 [ 0.929215] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.931278] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.933483] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.935263] PCI: CLS 0 bytes, default 64 [ 0.937110] Unpacking initramfs... [ 2.766445] debug: unmapping init [mem 0xffff9bb53cc54000-0xffff9bb53ffbffff] [ 2.770954] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.773987] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.777459] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.327950] Initialise system trusted keyrings [ 3.329526] Key type blacklist registered [ 3.331434] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.340567] zbud: loaded [ 3.342980] *** VALIDATE nfs *** [ 3.344209] *** VALIDATE nfs4 *** [ 3.345756] pstore: using deflate compression [ 3.349569] Platform Keyring initialized [ 3.459495] NET: Registered protocol family 38 [ 3.461539] Key type asymmetric registered [ 3.463142] Asymmetric key parser 'x509' registered [ 3.469121] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.472384] io scheduler mq-deadline registered [ 3.474207] io scheduler kyber registered [ 3.475990] io scheduler bfq registered [ 3.478247] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.481414] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.484342] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.488059] ACPI: Power Button [PWRF] [ 3.495488] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.503579] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.518469] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.532264] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.592202] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.624223] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.651844] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.780550] Non-volatile memory driver v1.3 [ 3.782229] Linux agpgart interface v0.103 [ 3.825356] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.828239] vda: detected capacity change from 0 to 68624384 [ 3.850908] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.854288] vdb: detected capacity change from 0 to 1073741824 [ 3.870492] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.873446] vdc: detected capacity change from 0 to 2621440000 [ 3.890952] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.894695] vdd: detected capacity change from 0 to 2621440000 [ 3.915625] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.919413] vde: detected capacity change from 0 to 4294967296 [ 3.978825] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.988358] vdf: detected capacity change from 0 to 4294967296 [ 4.031678] libphy: Fixed MDIO Bus: probed [ 4.037866] usbcore: registered new interface driver usbserial_generic [ 4.040466] usbserial: USB Serial support registered for generic [ 4.042714] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.047883] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.049778] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.052250] mousedev: PS/2 mouse device common for all mice [ 4.055692] rtc_cmos 00:05: RTC can wake from S4 [ 4.058785] rtc_cmos 00:05: registered as rtc0 [ 4.060216] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.064170] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.064227] intel_pstate: CPU model not supported [ 4.073411] hid: raw HID events driver (C) Jiri Kosina [ 4.079567] usbcore: registered new interface driver usbhid [ 4.081721] usbhid: USB HID core driver [ 4.083723] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.086084] drop_monitor: Initializing network drop monitor service [ 4.086204] Initializing XFRM netlink socket [ 4.087386] NET: Registered protocol family 10 [ 4.099645] Segment Routing with IPv6 [ 4.101531] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.102963] NET: Registered protocol family 17 [ 4.110367] mpls_gso: MPLS GSO support [ 4.116221] RAS: Correctable Errors collector initialized. [ 4.118522] AVX version of gcm_enc/dec engaged. [ 4.120429] AES CTR mode by8 optimization enabled [ 4.223812] sched_clock: Marking stable (4223795557, 0)->(5306493692, -1082698135) [ 4.241809] registered taskstats version 1 [ 4.260572] Loading compiled-in X.509 certificates [ 4.264145] zswap: loaded using pool lzo/zbud [ 4.325859] Key type big_key registered [ 4.342890] Key type encrypted registered [ 4.344743] ima: No TPM chip found, activating TPM-bypass! [ 4.348816] ima: Allocated hash algorithm: sha1 [ 4.350856] ima: No architecture policies found [ 4.353208] evm: Initialising EVM extended attributes: [ 4.355188] evm: security.selinux [ 4.356504] evm: security.ima [ 4.358387] evm: security.capability [ 4.360433] evm: HMAC attrs: 0x1 [ 4.364594] rtc_cmos 00:05: setting system clock to 2026-01-02 00:46:46 UTC (1767314806) [ 4.377469] debug: unmapping init [mem 0xffffffffb5003000-0xffffffffb51fffff] [ 4.382186] debug: unmapping init [mem 0xffffffffb3d82000-0xffffffffb4058fff] [ 4.397111] Write protecting the kernel read-only data: 28672k [ 4.402141] debug: unmapping init [mem 0xffffffffb2403000-0xffffffffb25fffff] [ 4.405357] debug: unmapping init [mem 0xffffffffb2d14000-0xffffffffb2dfffff] [ 4.452612] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.465475] systemd[1]: Detected virtualization kvm. [ 4.468118] systemd[1]: Detected architecture x86-64. [ 4.470614] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.507506] systemd[1]: No hostname configured. [ 4.511182] systemd[1]: Set hostname to . [ 4.514428] random: systemd: uninitialized urandom read (16 bytes read) [ 4.519834] systemd[1]: Initializing machine ID from random generator. [ 4.725397] random: ln: uninitialized urandom read (6 bytes read) [ 4.946084] random: systemd: uninitialized urandom read (16 bytes read) [ 4.950475] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.964613] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.983718] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.553553] device-mapper: uevent: version 1.0.3 [ 6.555517] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook...[ 7.603394] random: fast init done [ OK ] [ 7.678304] virtio_net virtio0 ens2: renamed from eth0 Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.129212] scsi host0: ata_piix [ 8.162900] scsi host1: ata_piix [ 8.164378] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.169613] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.890568] random: crng init done [ 12.895773] random: 7 urandom warning(s) missed due to ratelimiting [ 13.944523] dracut-initqueue[575]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.917155] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.712890] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.163803] SELinux: Disabled at runtime. [ 17.233983] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.245761] systemd[1]: Detected virtualization kvm. [ 17.247686] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.309635] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.313351] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.331373] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.335858] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.341232] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.351364] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.361975] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ 18.850773] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 19.718676] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.394330] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.499136] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.887686] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.955577] EDAC sbridge: Ver: 1.1.2 [ 24.475223] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit)[ 25.316721] NFS: Registering the id_resolver key type [ 25.322285] Key type id_resolver registered [ 25.326206] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg325-server login: [ 61.657918] libcfs: loading out-of-tree module taints kernel. [ 61.709745] Key type ._llcrypt registered [ 61.712053] Key type .llcrypt registered [ 61.804188] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_hostid [ 94.515464] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [ 95.147365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.155873] alg: No test for adler32 (adler32-zlib) [ 96.186254] Lustre: Lustre: Build Version: 2.17.0_RC4_5_g36e97ed [ 96.609888] LNet: Added LNI 192.168.203.125@tcp [8/256/0/180] [ 98.247990] Key type lgssc registered [ 99.222925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 116.515409] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 116.764050] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.973846] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.514961] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.514984] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 149.813894] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing load_modules_local [ 167.625247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.682798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.751784] Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 167.758571] Lustre: Skipped 1 previous similar message [ 167.776058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 167.779705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.974740] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 171.034966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 171.183910] Lustre: lustre-MDT0001: new disk, initializing [ 171.292596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.310924] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 171.320223] Lustre: 15469:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 171.326977] LustreError: 15469:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 171.362861] Lustre: 15452:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 171.717590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.771075] Lustre: 15499:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000004aef6ee x1853163986492800/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1767314990 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 174.836679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 174.843642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.524260] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.844263] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.507165] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.548693] Lustre: 16644:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 189.291615] hrtimer: interrupt took 2604637 ns [ 210.508752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 210.655303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 211.061775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 211.148196] Lustre: lustre-OST0000: new disk, initializing [ 211.154738] Lustre: Skipped 1 previous similar message [ 211.171825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 211.184383] Lustre: Skipped 2 previous similar messages [ 211.195412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 211.288774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.293504] Lustre: Skipped 1 previous similar message [ 219.384489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 219.404741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.474060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 219.506130] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 221.450379] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.581037] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.160429] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.143283] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.588764] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing check_logdir /tmp/testlogs/ [ 260.550416] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing yml_node [ 269.025614] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 271.918824] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 274.414112] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 276.378594] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Thu Jan 1 19:51:15 EST 2026 [ 293.667878] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 295.547680] Lustre: DEBUG MARKER: === replay-single: start setup 19:51:35 (1767315095) === [ 299.850198] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing check_config_client /mnt/lustre [ 350.699122] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 354.679063] Lustre: 24170:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 359.094629] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 364.026755] Lustre: DEBUG MARKER: === replay-single: finish setup 19:52:43 (1767315163) === [ 365.798512] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 19:52:45 (1767315165) [ 373.143158] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 375.161283] Lustre: Failing over lustre-MDT0000 [ 375.344515] LustreError: 25043:0:(obd_class.h:479:obd_check_dev()) Device 26 not setup [ 375.675193] Lustre: server umount lustre-MDT0000 complete [ 378.338435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.363566] Lustre: Skipped 2 previous similar messages [ 383.457572] LustreError: 15439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.474558] LustreError: 15439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 388.577673] LustreError: 15435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 388.610537] LustreError: 15435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 393.721237] LustreError: 18722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 393.756380] LustreError: 18722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 394.721304] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315180/real 1767315180] req@ffff9bb5aa26c380 x1853163986606336/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 394.753598] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 399.843394] LustreError: 15433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.872083] LustreError: 15433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 403.938858] LustreError: 15439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 403.957430] LustreError: 15439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 404.689544] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 404.693451] LDISKFS-fs (dm-0): recovery complete [ 404.708779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.178479] LustreError: 18722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 414.203116] LustreError: 18722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 419.297595] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb485c8c000 x1853163986619264/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 419.544369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.549503] Lustre: Skipped 1 previous similar message [ 419.707762] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 423.559884] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 424.936756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 424.990155] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 431.821129] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 433.485785] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 442.680729] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 19:54:01 (1767315241) [ 445.208159] Lustre: Failing over lustre-OST0000 [ 445.262119] LustreError: 26763:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 445.267866] LustreError: 26763:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 445.342984] Lustre: server umount lustre-OST0000 complete [ 445.348272] LustreError: 18316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.380777] LustreError: 18316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 445.415972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.416471] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 445.436434] Lustre: Skipped 2 previous similar messages [ 472.803914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 473.040336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.753623] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 474.979541] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 474.982335] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 474.999802] Lustre: Skipped 3 previous similar messages [ 482.858807] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 491.410882] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 493.541989] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 504.732519] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 19:55:04 (1767315304) [ 512.723624] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 515.476740] Lustre: Failing over lustre-MDT0000 [ 515.657910] LustreError: 28617:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 515.665381] LustreError: 28617:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 515.779148] Lustre: server umount lustre-MDT0000 complete [ 517.088495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 517.097155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.111104] LustreError: 15438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.124976] LustreError: 15438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 519.136744] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.153543] Lustre: Skipped 1 previous similar message [ 535.522590] Lustre: 6992:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315321/real 1767315321] req@ffff9bb4905f6a00 x1853163986680320/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315337 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 535.536797] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.564878] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 544.567492] LDISKFS-fs (dm-0): recovery complete [ 544.574258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 545.697388] LustreError: 29525:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 545.704268] LustreError: 29525:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb5883b4a80 x1853163986686080/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767315347 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 545.725522] LustreError: 29525:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 545.762094] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0dd8f0 [ 545.768030] Lustre: MGC192.168.203.125@tcp: Connection restored to 0@lo (at 0@lo) [ 545.773047] Lustre: Skipped 1 previous similar message [ 545.877971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.618605] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 550.889547] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 550.896588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 552.538460] Lustre: lustre-MDT0000: Denying connection for new client b1b066e9-c8ee-4277-9639-5d4b3959aa94 (at 192.168.203.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 557.960337] Lustre: lustre-MDT0000: Denying connection for new client b1b066e9-c8ee-4277-9639-5d4b3959aa94 (at 192.168.203.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 563.069852] Lustre: lustre-MDT0000: Denying connection for new client b1b066e9-c8ee-4277-9639-5d4b3959aa94 (at 192.168.203.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 568.214570] Lustre: lustre-MDT0000: Denying connection for new client b1b066e9-c8ee-4277-9639-5d4b3959aa94 (at 192.168.203.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 573.316529] Lustre: lustre-MDT0000: Denying connection for new client b1b066e9-c8ee-4277-9639-5d4b3959aa94 (at 192.168.203.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:47 [ 583.556900] Lustre: lustre-MDT0000: Denying connection for new client b1b066e9-c8ee-4277-9639-5d4b3959aa94 (at 192.168.203.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:36 [ 583.570924] Lustre: Skipped 1 previous similar message [ 604.037842] Lustre: lustre-MDT0000: Denying connection for new client b1b066e9-c8ee-4277-9639-5d4b3959aa94 (at 192.168.203.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:16 [ 604.066793] Lustre: Skipped 3 previous similar messages [ 620.503742] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 620.506426] Lustre: 29559:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 835d5caa-4b54-4a81-b931-abb1738bca2d@ [ 620.544277] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 620.577211] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 620.582064] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 620.598222] Lustre: Skipped 2 previous similar messages [ 620.631899] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:44 to 0x2c0000400:65) [ 620.646147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:65) [ 634.007821] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 19:57:13 (1767315433) [ 641.552974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 643.975653] Lustre: Failing over lustre-MDT0000 [ 644.138646] LustreError: 30619:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 644.149129] LustreError: 30619:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 644.294336] Lustre: server umount lustre-MDT0000 complete [ 646.111740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 646.120528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.133542] Lustre: Skipped 1 previous similar message [ 646.140409] LustreError: 26879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 646.168877] LustreError: 26879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 664.833733] Lustre: 6993:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315450/real 1767315450] req@ffff9bb586e2e680 x1853163986741376/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315466 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 664.873208] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 674.004375] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 674.005788] LDISKFS-fs (dm-0): recovery complete [ 674.010828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.170055] LustreError: 31529:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 675.186358] LustreError: 31529:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb582cdad80 x1853163986747264/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767315477 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 675.238036] LustreError: 31529:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 675.296451] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5a738d180 x1853163986749696/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 675.695537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 675.755630] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 680.929441] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 680.943389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 681.841376] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 684.865831] Lustre: lustre-MDT0000: Denying connection for new client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 684.884642] Lustre: Skipped 3 previous similar messages [ 750.508824] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 750.519675] Lustre: 31563:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client b1b066e9-c8ee-4277-9639-5d4b3959aa94@ [ 750.541546] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 750.564528] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 750.564963] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 750.585851] Lustre: Skipped 2 previous similar messages [ 750.633087] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:44 to 0x2c0000400:97) [ 750.636474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:97) [ 762.143717] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 19:59:21 (1767315561) [ 770.919539] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 772.951238] Lustre: Failing over lustre-MDT0000 [ 773.090209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.094935] LustreError: 32623:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 773.099187] Lustre: Skipped 3 previous similar messages [ 773.099858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.109578] LustreError: 32623:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 773.379353] Lustre: server umount lustre-MDT0000 complete [ 777.112587] LustreError: 15434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 777.144253] LustreError: 15434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 794.596096] Lustre: 6993:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315580/real 1767315580] req@ffff9bb5812ae680 x1853163986798976/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 794.618285] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 803.452628] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 803.458797] LDISKFS-fs (dm-0): recovery complete [ 803.468418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 804.639128] LustreError: 33533:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 804.645155] LustreError: 33533:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb5812afb80 x1853163986804224/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767315606 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 804.657276] LustreError: 33533:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 804.832635] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb582cd9180 x1853163986806912/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 805.110655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.153981] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 806.280963] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 810.093292] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 810.469450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 810.543301] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 810.570288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:129) [ 810.571117] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:44 to 0x2c0000400:129) [ 819.338494] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 821.464854] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 837.934925] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 20:00:36 (1767315636) [ 846.401712] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 848.292125] Lustre: Failing over lustre-MDT0000 [ 848.442859] LustreError: 34742:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 848.448932] LustreError: 34742:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 848.599754] Lustre: server umount lustre-MDT0000 complete [ 851.434463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 851.436689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.452492] Lustre: Skipped 5 previous similar messages [ 867.809083] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315653/real 1767315653] req@ffff9bb5ae3e2d80 x1853163986839424/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315669 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 867.845564] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 880.822262] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 880.831208] LDISKFS-fs (dm-0): recovery complete [ 880.852878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.391052] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0de68e [ 892.411574] Lustre: MGC192.168.203.125@tcp: Connection restored to 0@lo (at 0@lo) [ 892.417745] Lustre: Skipped 3 previous similar messages [ 892.672077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 892.739800] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 892.807184] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 898.177500] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 898.246798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:44 to 0x280000400:161) [ 898.249064] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:131 to 0x2c0000400:161) [ 898.901805] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 907.746298] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 909.581135] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 920.139143] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 20:01:59 (1767315719) [ 929.163442] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 931.846660] Lustre: Failing over lustre-MDT0000 [ 932.040461] LustreError: 36876:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 932.053915] LustreError: 36876:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 932.304793] Lustre: server umount lustre-MDT0000 complete [ 933.859520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.879988] Lustre: Skipped 3 previous similar messages [ 949.151102] Lustre: 6994:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315735/real 1767315735] req@ffff9bb5b326b100 x1853163986880384/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315751 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 949.195307] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 964.987622] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 964.990795] LDISKFS-fs (dm-0): recovery complete [ 965.015995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.167662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.237031] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 976.448657] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 980.359504] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 980.626445] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 980.700789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 980.702526] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:131 to 0x2c0000400:193) [ 989.188699] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 990.728617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 999.967120] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 20:03:19 (1767315799) [ 1008.067280] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1010.351106] Lustre: Failing over lustre-MDT0000 [ 1010.551663] LustreError: 38993:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1010.564869] LustreError: 38993:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1010.713612] Lustre: server umount lustre-MDT0000 complete [ 1011.170688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1011.188069] Lustre: Skipped 2 previous similar messages [ 1027.555694] Lustre: 6992:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315813/real 1767315813] req@ffff9bb5bcddf480 x1853163986919424/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1027.607708] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1034.113410] LustreError: 15434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1034.127697] LustreError: 15434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 156 previous similar messages [ 1040.912415] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1040.914742] LDISKFS-fs (dm-0): recovery complete [ 1040.921627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.137794] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0df100 [ 1052.160095] Lustre: MGC192.168.203.125@tcp: Connection restored to 0@lo (at 0@lo) [ 1052.170435] Lustre: Skipped 8 previous similar messages [ 1052.511187] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1054.590728] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1057.551612] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1057.936213] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1057.971535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 1057.974163] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 1065.606921] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1067.040295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1076.568048] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 20:04:35 (1767315875) [ 1085.125882] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1087.046329] Lustre: Failing over lustre-MDT0000 [ 1087.219509] LustreError: 41112:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1087.223771] LustreError: 41112:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1087.329512] Lustre: server umount lustre-MDT0000 complete [ 1104.927249] Lustre: 6994:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315890/real 1767315890] req@ffff9bb584de5f80 x1853163986959872/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315906 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1104.971449] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.580320] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1117.582644] LDISKFS-fs (dm-0): recovery complete [ 1117.588470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1130.017854] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5b326ad80 x1853163986973824/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1130.564090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.578690] Lustre: Skipped 1 previous similar message [ 1130.668386] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1131.418293] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1135.704786] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1135.771795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:257) [ 1135.779526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:257) [ 1136.527074] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1145.632185] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1147.677782] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1159.073143] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 20:05:58 (1767315958) [ 1160.148784] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1160.152665] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1160.155787] LustreError: 15436:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb58907fb80 x1853163952043008/t38654705666(0) o35->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:648/0 lens 392/456 e 0 to 0 dl 1767315973 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1170.108818] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1172.482235] Lustre: Failing over lustre-MDT0000 [ 1172.865665] Lustre: server umount lustre-MDT0000 complete [ 1176.568470] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1176.580991] Lustre: Skipped 8 previous similar messages [ 1191.906961] Lustre: 6992:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315978/real 1767315978] req@ffff9bb5890a8000 x1853163987006848/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1191.947597] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1202.195108] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1202.197358] LDISKFS-fs (dm-0): recovery complete [ 1202.217848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.506834] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5b2825180 x1853163987019008/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1217.949147] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1223.086350] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1223.243228] Lustre: 15436:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5894d4380 x1853163952043008/t38654705666(0) o35->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:711/0 lens 392/456 e 0 to 0 dl 1767316036 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1223.264464] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:289) [ 1223.274643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:289) [ 1231.017441] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1232.885384] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1242.418334] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 20:07:21 (1767316041) [ 1251.728670] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1253.999077] Lustre: Failing over lustre-MDT0000 [ 1254.194781] LustreError: 45402:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1254.211617] LustreError: 45402:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 1254.353683] Lustre: server umount lustre-MDT0000 complete [ 1258.985035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1287.434439] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1287.438893] LDISKFS-fs (dm-0): recovery complete [ 1287.452657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.346662] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1300.354423] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1300.380461] Lustre: Skipped 1 previous similar message [ 1305.702957] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1305.711931] Lustre: Skipped 1 previous similar message [ 1305.772093] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:321) [ 1305.772558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:321) [ 1306.576236] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1315.920546] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1318.123617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1328.405843] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 20:08:47 (1767316127) [ 1338.164191] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1339.246645] Lustre: *** cfs_fail_loc=114, val=0*** [ 1342.700200] Lustre: Failing over lustre-MDT0000 [ 1343.224609] Lustre: server umount lustre-MDT0000 complete [ 1361.874290] Lustre: 6993:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316148/real 1767316148] req@ffff9bb4839f3800 x1853163987089280/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1361.920523] Lustre: 6993:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1361.926903] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.944703] LustreError: Skipped 1 previous similar message [ 1375.691628] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1375.696980] LDISKFS-fs (dm-0): recovery complete [ 1375.721690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.488610] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5aa31c000 x1853163987100288/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1387.745940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.749627] Lustre: Skipped 2 previous similar messages [ 1387.790740] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1393.135531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1393.155374] Lustre: Skipped 16 previous similar messages [ 1393.265076] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:353) [ 1393.270063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:353) [ 1393.424326] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1401.698403] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1403.247632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1412.667566] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 20:10:11 (1767316211) [ 1420.885992] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1422.063256] Lustre: *** cfs_fail_loc=128, val=0*** [ 1425.417049] Lustre: Failing over lustre-MDT0000 [ 1425.834973] Lustre: server umount lustre-MDT0000 complete [ 1454.263858] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1454.268100] LDISKFS-fs (dm-0): recovery complete [ 1454.280652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1455.455127] LustreError: 50739:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1455.465831] LustreError: 50739:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb48575bb80 x1853163987134976/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767316257 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1455.491775] LustreError: 50739:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1455.596542] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0e0980 [ 1460.330461] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1461.362644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:385) [ 1461.369351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:385) [ 1468.607932] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1470.604536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1479.528333] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 20:11:19 (1767316279) [ 1488.453405] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1491.782080] Lustre: Failing over lustre-MDT0000 [ 1491.858780] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.25@tcp (stopping) [ 1491.873195] Lustre: Skipped 3 previous similar messages [ 1491.938719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1491.944701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1491.957726] Lustre: Skipped 14 previous similar messages [ 1492.198985] Lustre: server umount lustre-MDT0000 complete [ 1523.074300] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1523.076627] LDISKFS-fs (dm-0): recovery complete [ 1523.081150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1523.684549] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb4860e7480 x1853163987174656/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1523.927647] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1523.938343] Lustre: Skipped 1 previous similar message [ 1527.986196] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1529.727325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:391 to 0x280000400:417) [ 1529.727666] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:391 to 0x2c0000400:417) [ 1535.239787] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1536.846862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1545.434359] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 20:12:25 (1767316345) [ 1554.241068] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1556.144890] Lustre: Failing over lustre-MDT0000 [ 1556.315126] LustreError: 54065:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 1556.321511] LustreError: 54065:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 1556.437127] Lustre: server umount lustre-MDT0000 complete [ 1558.407592] LustreError: 15433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.432529] LustreError: 15433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 287 previous similar messages [ 1586.333727] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1586.336825] LDISKFS-fs (dm-0): recovery complete [ 1586.351251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.628509] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1589.650268] Lustre: Skipped 3 previous similar messages [ 1592.229796] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1592.583409] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1592.594092] Lustre: Skipped 3 previous similar messages [ 1592.658544] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:423 to 0x2c0000400:449) [ 1592.662833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:423 to 0x280000400:449) [ 1601.655617] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1603.611109] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1611.729446] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 20:13:31 (1767316411) [ 1620.609376] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1628.394463] Lustre: Failing over lustre-MDT0000 [ 1628.647417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1628.866600] Lustre: server umount lustre-MDT0000 complete [ 1649.635462] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316435/real 1767316435] req@ffff9bb586e2d880 x1853163987243392/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1649.666557] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1649.676683] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1649.684515] LustreError: Skipped 3 previous similar messages [ 1659.477425] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1659.481670] LDISKFS-fs (dm-0): recovery complete [ 1659.490181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.873031] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5bd77f100 x1853163987251712/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1665.071164] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1669.621275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:560 to 0x2c0000400:577) [ 1669.624411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:577) [ 1673.937934] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1675.439503] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1702.351914] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 20:15:01 (1767316501) [ 1711.331977] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1713.611256] Lustre: Failing over lustre-MDT0000 [ 1714.082327] Lustre: server umount lustre-MDT0000 complete [ 1715.698922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1746.322163] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1746.331231] LDISKFS-fs (dm-0): recovery complete [ 1746.339913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.664742] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb582cda680 x1853163987356288/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1762.960332] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1763.489370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:560 to 0x2c0000400:609) [ 1763.494058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:609) [ 1772.177513] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1773.980395] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1785.952385] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 20:16:25 (1767316585) [ 1794.602669] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1796.797683] Lustre: Failing over lustre-MDT0000 [ 1797.217888] Lustre: server umount lustre-MDT0000 complete [ 1827.271722] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1827.273554] LDISKFS-fs (dm-0): recovery complete [ 1827.291406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.377685] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1841.385935] Lustre: Skipped 3 previous similar messages [ 1846.494377] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1846.885100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:641) [ 1846.888423] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:560 to 0x2c0000400:641) [ 1856.110454] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1858.132055] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1866.729064] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 20:17:46 (1767316666) [ 1874.979896] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1876.801204] Lustre: Failing over lustre-MDT0000 [ 1877.272720] Lustre: server umount lustre-MDT0000 complete [ 1877.472893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1907.112508] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1907.115200] LDISKFS-fs (dm-0): recovery complete [ 1907.129608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.450967] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5b9833480 x1853163987439872/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1918.701792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.707830] Lustre: Skipped 6 previous similar messages [ 1923.642208] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1924.091192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1924.102069] Lustre: Skipped 28 previous similar messages [ 1924.272933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:673) [ 1924.273412] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:560 to 0x2c0000400:673) [ 1932.466815] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1934.596834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1944.495351] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 20:19:04 (1767316744) [ 1952.559303] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1954.588986] Lustre: Failing over lustre-MDT0000 [ 1954.849228] Lustre: server umount lustre-MDT0000 complete [ 1987.033790] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1987.035678] LDISKFS-fs (dm-0): recovery complete [ 1987.069042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2001.382120] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0ec6c6 [ 2007.207812] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:560 to 0x2c0000400:705) [ 2007.207860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:705) [ 2007.457587] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2014.467264] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2016.153800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2024.369095] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 20:20:23 (1767316823) [ 2032.969831] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2035.388182] Lustre: Failing over lustre-MDT0000 [ 2035.838874] Lustre: server umount lustre-MDT0000 complete [ 2037.733098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.775321] Lustre: Skipped 27 previous similar messages [ 2066.639693] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2066.643964] LDISKFS-fs (dm-0): recovery complete [ 2066.650931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.695129] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0ecbce [ 2083.820276] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2084.457668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:737) [ 2084.465229] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:560 to 0x2c0000400:737) [ 2091.830616] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2093.341689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2101.899817] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 20:21:41 (1767316901) [ 2109.148942] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2110.923356] Lustre: Failing over lustre-MDT0000 [ 2111.072412] LustreError: 68930:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2111.078260] LustreError: 68930:0:(obd_class.h:479:obd_check_dev()) Skipped 55 previous similar messages [ 2111.177756] Lustre: server umount lustre-MDT0000 complete [ 2115.042603] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2138.983232] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2138.985553] LDISKFS-fs (dm-0): recovery complete [ 2138.995398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.221091] LustreError: 69844:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2141.238524] LustreError: 69844:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb5b9830e00 x1853163987549568/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767316943 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2141.286183] LustreError: 69844:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2143.044969] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2143.056065] Lustre: Skipped 6 previous similar messages [ 2147.098258] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2147.398393] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2147.402967] Lustre: Skipped 6 previous similar messages [ 2147.450295] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:560 to 0x2c0000400:769) [ 2147.457890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:560 to 0x280000400:769) [ 2155.064840] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2157.099233] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2165.310382] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 20:22:44 (1767316964) [ 2174.113440] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2176.545627] Lustre: Failing over lustre-MDT0000 [ 2177.005881] Lustre: server umount lustre-MDT0000 complete [ 2178.019129] LustreError: 15435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.038308] LustreError: 15435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 355 previous similar messages [ 2194.404980] Lustre: 6994:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316980/real 1767316980] req@ffff9bb5a7f98a80 x1853163987578496/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2194.437562] Lustre: 6994:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 2194.443575] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2194.455699] LustreError: Skipped 6 previous similar messages [ 2208.861687] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2208.864610] LDISKFS-fs (dm-0): recovery complete [ 2208.880637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.813956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:771 to 0x280000400:801) [ 2224.820935] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:771 to 0x2c0000400:801) [ 2226.551707] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2235.799299] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2237.749033] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2247.043576] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 20:24:06 (1767317046) [ 2255.354223] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2257.486335] Lustre: Failing over lustre-MDT0000 [ 2257.720557] Lustre: server umount lustre-MDT0000 complete [ 2287.884674] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2287.891768] LDISKFS-fs (dm-0): recovery complete [ 2287.921070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2301.412303] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0edcd7 [ 2307.130573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:771 to 0x280000400:833) [ 2307.133323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:771 to 0x2c0000400:833) [ 2307.342426] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2315.748770] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2317.388459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2326.653707] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 20:25:26 (1767317126) [ 2334.813848] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2336.783444] Lustre: Failing over lustre-MDT0000 [ 2337.057527] Lustre: server umount lustre-MDT0000 complete [ 2366.139701] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2366.144097] LDISKFS-fs (dm-0): recovery complete [ 2366.159953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.734092] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5b9831880 x1853163987671936/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2378.765349] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 2379.105722] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2379.121458] Lustre: Skipped 6 previous similar messages [ 2383.680306] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2384.433702] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:771 to 0x2c0000400:865) [ 2384.440657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:835 to 0x280000400:865) [ 2393.353428] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2395.793307] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2404.731508] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 20:26:44 (1767317204) [ 2412.895779] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2414.768212] Lustre: Failing over lustre-MDT0000 [ 2415.076745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2415.082672] LustreError: Skipped 1 previous similar message [ 2415.099541] Lustre: server umount lustre-MDT0000 complete [ 2445.467172] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2445.469588] LDISKFS-fs (dm-0): recovery complete [ 2445.488065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.791304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:867 to 0x280000400:897) [ 2461.792727] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:771 to 0x2c0000400:897) [ 2461.960661] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2470.540659] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2472.814485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2482.269432] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 20:28:01 (1767317281) [ 2490.489961] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2492.609523] Lustre: Failing over lustre-MDT0000 [ 2492.960496] Lustre: server umount lustre-MDT0000 complete [ 2520.918639] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2520.923391] LDISKFS-fs (dm-0): recovery complete [ 2520.937852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.146380] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0eeceb [ 2524.159260] Lustre: MGC192.168.203.125@tcp: Connection restored to 0@lo (at 0@lo) [ 2524.177656] Lustre: Skipped 34 previous similar messages [ 2524.527429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.530137] Lustre: Skipped 7 previous similar messages [ 2529.912061] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:899 to 0x2c0000400:929) [ 2529.912596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:899 to 0x280000400:929) [ 2530.164236] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2538.304597] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2539.986795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2548.271939] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 20:29:07 (1767317347) [ 2556.653782] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2558.988318] Lustre: Failing over lustre-MDT0000 [ 2559.374735] Lustre: server umount lustre-MDT0000 complete [ 2588.788938] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2588.791334] LDISKFS-fs (dm-0): recovery complete [ 2588.799874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.928716] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2608.286674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:899 to 0x280000400:961) [ 2608.286713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:931 to 0x2c0000400:961) [ 2616.718855] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2618.669160] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2628.104515] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 20:30:27 (1767317427) [ 2636.313966] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2638.594341] Lustre: Failing over lustre-MDT0000 [ 2638.823621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2638.837804] Lustre: Skipped 30 previous similar messages [ 2638.936692] Lustre: server umount lustre-MDT0000 complete [ 2668.677745] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2668.681032] LDISKFS-fs (dm-0): recovery complete [ 2668.716238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.047190] LustreError: 84674:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2670.059587] LustreError: 84674:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb5a7f1df80 x1853163987822080/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767317472 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2670.079141] LustreError: 84674:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2670.576126] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0ef79c [ 2676.173792] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2676.261102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:899 to 0x280000400:993) [ 2676.262161] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:963 to 0x2c0000400:993) [ 2684.450456] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2686.268102] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2695.249664] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 20:31:35 (1767317495) [ 2702.403599] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2704.430817] Lustre: Failing over lustre-MDT0000 [ 2704.717635] Lustre: server umount lustre-MDT0000 complete [ 2734.094130] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2734.096103] LDISKFS-fs (dm-0): recovery complete [ 2734.108958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.880689] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2748.895961] Lustre: Skipped 7 previous similar messages [ 2753.127973] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2753.646379] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2753.659130] Lustre: Skipped 7 previous similar messages [ 2753.721126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:995 to 0x280000400:1025) [ 2753.723891] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:995 to 0x2c0000400:1025) [ 2761.578891] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2763.149258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2771.899262] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 20:32:51 (1767317571) [ 2778.976934] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2781.115775] Lustre: Failing over lustre-MDT0000 [ 2781.300285] LustreError: 88001:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2781.315478] LustreError: 88001:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 2781.526412] Lustre: server umount lustre-MDT0000 complete [ 2784.138859] LustreError: 15433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2784.158241] LustreError: 15433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 360 previous similar messages [ 2800.612363] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317586/real 1767317586] req@ffff9bb5b3839c00 x1853163987891072/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767317602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2800.642910] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2800.652728] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2800.659891] LustreError: Skipped 7 previous similar messages [ 2809.798408] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2809.800408] LDISKFS-fs (dm-0): recovery complete [ 2809.806717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.524625] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2816.716289] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1027 to 0x2c0000400:1057) [ 2816.717106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1027 to 0x280000400:1057) [ 2823.506516] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2825.495790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2834.548790] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:33:54 (1767317634) [ 2842.579330] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2844.533948] Lustre: Failing over lustre-MDT0000 [ 2845.009525] Lustre: server umount lustre-MDT0000 complete [ 2873.060919] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2873.068472] LDISKFS-fs (dm-0): recovery complete [ 2873.076388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2873.832464] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0f093f [ 2873.843862] Lustre: Skipped 1 previous similar message [ 2878.777053] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2879.599357] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1059 to 0x2c0000400:1089) [ 2879.599460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1027 to 0x280000400:1089) [ 2886.116389] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2887.883588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2895.985517] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 20:34:55 (1767317695) [ 2900.204384] Lustre: 91880:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b170ef68-e829-4c28-bf9f-1c545bc4c72c at adminstrative request [ 2907.227211] Lustre: Failing over lustre-MDT0000 [ 2907.446666] Lustre: server umount lustre-MDT0000 complete [ 2931.941962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.804100] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5a663ca80 x1853163987972608/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2936.829230] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 11 previous similar messages [ 2942.220966] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2942.520642] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1059 to 0x2c0000400:1121) [ 2942.522081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1091 to 0x280000400:1121) [ 2950.045466] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2951.437827] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2957.635301] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2964.730333] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2971.579173] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 20:36:10 (1767317770) [ 2972.745886] Lustre: 93924:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b170ef68-e829-4c28-bf9f-1c545bc4c72c at adminstrative request [ 2981.816721] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:36:21 (1767317781) [ 2990.541189] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2992.957565] Lustre: Failing over lustre-MDT0000 [ 2993.310764] Lustre: server umount lustre-MDT0000 complete [ 2993.635975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2993.647718] LustreError: Skipped 3 previous similar messages [ 3023.072829] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3023.074498] LDISKFS-fs (dm-0): recovery complete [ 3023.095397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.954300] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3034.970200] Lustre: Skipped 8 previous similar messages [ 3039.434807] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3040.510547] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1124 to 0x2c0000400:1153) [ 3040.510838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1123 to 0x280000400:1153) [ 3047.872222] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3049.709983] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3059.319682] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:37:38 (1767317858) [ 3067.640870] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3069.509273] Lustre: Failing over lustre-MDT0000 [ 3071.684852] Lustre: server umount lustre-MDT0000 complete [ 3099.485079] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3099.488071] LDISKFS-fs (dm-0): recovery complete [ 3099.495813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.167221] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3117.645119] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1155 to 0x2c0000400:1185) [ 3117.645771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1155 to 0x280000400:1185) [ 3125.360748] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3127.241075] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3136.539816] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:38:55 (1767317935) [ 3145.903482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3148.007779] Lustre: Failing over lustre-MDT0000 [ 3148.297768] Lustre: server umount lustre-MDT0000 complete [ 3179.450885] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3179.453287] LDISKFS-fs (dm-0): recovery complete [ 3179.490712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.649477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3194.660667] Lustre: Skipped 8 previous similar messages [ 3199.409958] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3199.996943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3200.001406] Lustre: Skipped 40 previous similar messages [ 3200.097302] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1187 to 0x2c0000400:1217) [ 3200.105511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1187 to 0x280000400:1217) [ 3208.245057] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3210.185309] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3219.149799] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 20:40:18 (1767318018) [ 3227.931152] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3229.914151] Lustre: Failing over lustre-MDT0000 [ 3232.178900] Lustre: server umount lustre-MDT0000 complete [ 3261.798323] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3261.801475] LDISKFS-fs (dm-0): recovery complete [ 3261.808199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3276.376310] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3276.856139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1219 to 0x2c0000400:1249) [ 3276.858262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1219 to 0x280000400:1249) [ 3284.205935] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3285.958747] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3294.523928] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:41:33 (1767318093) [ 3301.982679] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3304.007227] Lustre: Failing over lustre-MDT0000 [ 3304.321064] Lustre: server umount lustre-MDT0000 complete [ 3307.488029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3307.500547] Lustre: Skipped 34 previous similar messages [ 3332.879981] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3332.882153] LDISKFS-fs (dm-0): recovery complete [ 3332.891530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.048734] LustreError: 103990:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 3334.058397] LustreError: 103990:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb5b3300000 x1853163988173952/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767318136 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3334.087644] LustreError: 103990:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 3338.591579] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3339.842404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1219 to 0x280000400:1281) [ 3339.843286] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1251 to 0x2c0000400:1281) [ 3347.071089] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3348.895929] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3358.017684] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:42:37 (1767318157) [ 3365.918422] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3368.177033] Lustre: Failing over lustre-MDT0000 [ 3368.427246] Lustre: server umount lustre-MDT0000 complete [ 3385.823890] LustreError: 15434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.866813] LustreError: 15434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 348 previous similar messages [ 3398.234993] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3398.244134] LDISKFS-fs (dm-0): recovery complete [ 3398.267084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.427938] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d0f2f48 [ 3411.431951] Lustre: Skipped 1 previous similar message [ 3412.352884] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3412.370412] Lustre: Skipped 8 previous similar messages [ 3417.199265] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3417.218775] Lustre: Skipped 8 previous similar messages [ 3417.275488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1283 to 0x280000400:1313) [ 3417.280359] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1283 to 0x2c0000400:1313) [ 3417.369839] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3427.903312] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3430.506789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3443.339401] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:44:02 (1767318242) [ 3451.708955] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3453.836693] Lustre: Failing over lustre-MDT0000 [ 3454.013243] LustreError: 107316:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3454.026316] LustreError: 107316:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 3454.545922] Lustre: server umount lustre-MDT0000 complete [ 3474.399202] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318260/real 1767318260] req@ffff9bb5a7e0ed80 x1853163988248192/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767318276 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3474.445703] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 3474.456160] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3474.476956] LustreError: Skipped 8 previous similar messages [ 3484.280820] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3484.282785] LDISKFS-fs (dm-0): recovery complete [ 3484.296684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.290527] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3491.508762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1315 to 0x280000400:1345) [ 3491.522041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1315 to 0x2c0000400:1345) [ 3500.712549] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3502.795248] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3513.561857] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:45:12 (1767318312) [ 3522.477607] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3524.927700] Lustre: Failing over lustre-MDT0000 [ 3525.295217] Lustre: server umount lustre-MDT0000 complete [ 3555.197272] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3555.199114] LDISKFS-fs (dm-0): recovery complete [ 3555.210280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.471789] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3573.869542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1347 to 0x2c0000400:1377) [ 3573.869757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1347 to 0x280000400:1377) [ 3582.033996] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3583.928891] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3592.718694] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:46:32 (1767318392) [ 3601.509558] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3603.637209] Lustre: Failing over lustre-MDT0000 [ 3603.859819] Lustre: server umount lustre-MDT0000 complete [ 3604.460974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3604.466113] LustreError: Skipped 2 previous similar messages [ 3634.495787] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3634.498189] LDISKFS-fs (dm-0): recovery complete [ 3634.508928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3645.753530] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3645.767560] Lustre: Skipped 7 previous similar messages [ 3650.886673] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3651.206662] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1379 to 0x2c0000400:1409) [ 3651.207202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1379 to 0x280000400:1409) [ 3660.131372] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3662.185345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3672.384931] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:47:51 (1767318471) [ 3681.012372] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3683.277748] Lustre: Failing over lustre-MDT0000 [ 3683.540501] Lustre: server umount lustre-MDT0000 complete [ 3712.579789] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3712.582314] LDISKFS-fs (dm-0): recovery complete [ 3712.592624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.214979] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3719.233985] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1411 to 0x2c0000400:1441) [ 3719.237633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1411 to 0x280000400:1441) [ 3725.713474] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3727.771727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3736.864509] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 20:48:56 (1767318536) [ 3746.196204] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3748.589888] Lustre: Failing over lustre-MDT0000 [ 3750.837642] Lustre: server umount lustre-MDT0000 complete [ 3780.687119] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3780.689840] LDISKFS-fs (dm-0): recovery complete [ 3780.698627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3790.823342] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5a6634000 x1853163988413696/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3790.886437] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 52 previous similar messages [ 3796.613894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1443 to 0x2c0000400:1473) [ 3796.617432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1443 to 0x280000400:1473) [ 3796.861974] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3807.984432] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3810.391267] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3821.755558] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 20:50:20 (1767318620) [ 3822.624387] Lustre: 117455:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b170ef68-e829-4c28-bf9f-1c545bc4c72c at adminstrative request [ 3834.000690] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 20:50:33 (1767318633) [ 3842.571858] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3844.974642] Lustre: Failing over lustre-MDT0000 [ 3845.341471] Lustre: server umount lustre-MDT0000 complete [ 3863.571506] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3863.576511] LDISKFS-fs (dm-0): recovery complete [ 3863.586827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.254357] Lustre: MGC192.168.203.125@tcp: Connection restored to 0@lo (at 0@lo) [ 3873.258404] Lustre: Skipped 36 previous similar messages [ 3873.414927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.420838] Lustre: Skipped 8 previous similar messages [ 3873.457836] Lustre: lustre-MDT0000: Aborting client recovery [ 3873.461690] LustreError: 118884:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3873.469304] Lustre: 118920:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3873.477875] Lustre: 118920:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client b170ef68-e829-4c28-bf9f-1c545bc4c72c@ [ 3873.487804] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3873.493563] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3873.505836] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000400:0x1:0x0] [ 3873.587038] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1475 to 0x2c0000400:1505) [ 3873.591203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1443 to 0x280000400:1505) [ 3878.285877] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3878.886310] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3900.581647] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 20:51:40 (1767318700) [ 3909.333968] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3911.440404] Lustre: Failing over lustre-MDT0000 [ 3911.744729] Lustre: server umount lustre-MDT0000 complete [ 3914.740801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.764892] Lustre: Skipped 31 previous similar messages [ 3928.955985] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3928.957878] LDISKFS-fs (dm-0): recovery complete [ 3928.978638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.408372] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3929.462766] Lustre: lustre-MDT0000: Aborting client recovery [ 3929.466890] LustreError: 120880:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3929.478160] Lustre: 120913:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3929.495766] Lustre: 120913:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3929.502893] Lustre: 120913:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3929.509900] Lustre: 120913:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3929.518911] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3929.539798] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3929.590531] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000402:0x1:0x0] [ 3929.646846] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1511 to 0x2c0000400:1537) [ 3929.653816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1511 to 0x280000400:1537) [ 3934.703091] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3935.082245] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3949.471177] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3949.473864] Lustre: Skipped 1 previous similar message [ 3957.132491] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 20:52:36 (1767318756) [ 3966.202819] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3968.196241] Lustre: Failing over lustre-MDT0000 [ 3968.576592] Lustre: server umount lustre-MDT0000 complete [ 3985.898306] LustreError: 15433:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3985.916370] LustreError: 15433:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 312 previous similar messages [ 3986.190721] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3986.194344] LDISKFS-fs (dm-0): recovery complete [ 3986.203589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.468754] Lustre: lustre-MDT0000: Aborting client recovery [ 3996.472717] LustreError: 122877:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3996.484140] Lustre: 122912:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3996.498827] Lustre: 122912:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3996.510027] Lustre: 122912:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3996.520713] Lustre: 122912:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3996.528730] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3996.536898] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3996.567507] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3996.623535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1543 to 0x280000400:1569) [ 3996.638490] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1544 to 0x2c0000400:1569) [ 4001.473221] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4001.776510] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4025.798559] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 20:53:44 (1767318824) [ 4027.323823] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4027.330081] LustreError: 15434:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb584139880 x1853163953169024/t201863462916(0) o36->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:495/0 lens 512/456 e 0 to 0 dl 1767318840 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 4031.390128] Lustre: Failing over lustre-MDT0000 [ 4033.716283] Lustre: server umount lustre-MDT0000 complete [ 4047.643845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.137884] Lustre: lustre-MDT0000: Aborting client recovery [ 4048.141587] LustreError: 124334:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4048.146406] Lustre: 124367:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4048.149445] Lustre: 124367:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 4048.155591] Lustre: 124367:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4048.162267] Lustre: 124367:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4048.167376] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4048.174945] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 4048.211184] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 4048.302881] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1571 to 0x2c0000400:1601) [ 4048.311847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1543 to 0x280000400:1601) [ 4052.390353] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4053.489338] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4074.165889] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 4076.135710] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 20:54:35 (1767318875) [ 4085.831980] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4089.113749] Lustre: Failing over lustre-MDT0000 [ 4089.252162] LustreError: 125638:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4089.258386] LustreError: 125638:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 4089.312778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4089.317580] Lustre: Skipped 6 previous similar messages [ 4089.464273] Lustre: server umount lustre-MDT0000 complete [ 4105.454033] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4105.457985] LDISKFS-fs (dm-0): recovery complete [ 4105.465249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4105.619435] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4105.635944] LustreError: Skipped 8 previous similar messages [ 4106.071399] Lustre: lustre-MDT0000: Aborting client recovery [ 4106.076440] LustreError: 126428:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4106.082732] Lustre: 126461:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4106.098531] Lustre: 126461:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 4106.104287] Lustre: 126461:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4106.111675] Lustre: 126461:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4106.116667] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4106.122641] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 4106.134381] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 4106.225537] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:1571 to 0x2c0000400:1633) [ 4106.262636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:1543 to 0x280000400:1633) [ 4111.347154] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4112.241489] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4137.118231] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:55:36 (1767318936) [ 4173.889145] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4175.931763] Lustre: Failing over lustre-MDT0000 [ 4176.533653] Lustre: server umount lustre-MDT0000 complete [ 4194.271120] Lustre: 6994:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318980/real 1767318980] req@ffff9bb5b9b60a80 x1853163988697216/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767318996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4194.308033] Lustre: 6994:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 4204.015445] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4204.022677] LDISKFS-fs (dm-0): recovery complete [ 4204.030972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.485493] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4207.490468] Lustre: Skipped 5 previous similar messages [ 4209.413387] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4210.325650] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4210.337642] Lustre: Skipped 5 previous similar messages [ 4210.395703] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2034 to 0x2c0000400:2049) [ 4210.403306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2034 to 0x280000400:2049) [ 4219.247510] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4221.668065] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4253.330969] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:57:32 (1767319052) [ 4280.959969] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4291.709880] Lustre: Failing over lustre-MDT0000 [ 4292.261836] Lustre: server umount lustre-MDT0000 complete [ 4321.455745] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4321.461453] LDISKFS-fs (dm-0): recovery complete [ 4321.473244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4333.029569] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d11bab8 [ 4333.048518] Lustre: Skipped 1 previous similar message [ 4333.342363] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4333.351446] Lustre: Skipped 18 previous similar messages [ 4338.198755] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4342.382188] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2450 to 0x2c0000400:2465) [ 4342.383900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2450 to 0x280000400:2465) [ 4346.533889] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4348.791566] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4372.376130] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 20:59:31 (1767319171) [ 4374.908605] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4376.108083] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4376.119119] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4383.896634] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 20:59:43 (1767319183) [ 4411.538385] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4422.970853] Lustre: Failing over lustre-OST0000 [ 4423.041391] Lustre: server umount lustre-OST0000 complete [ 4424.672979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4424.680947] LustreError: Skipped 6 previous similar messages [ 4451.642618] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 4451.645858] LDISKFS-fs (dm-2): recovery complete [ 4451.660851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4459.366737] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4520.321340] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 21:01:59 (1767319319) [ 4528.505706] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4531.470904] Lustre: Failing over lustre-MDT0000 [ 4531.703258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4531.723303] Lustre: Skipped 25 previous similar messages [ 4531.924945] Lustre: server umount lustre-MDT0000 complete [ 4561.930181] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4561.936077] LDISKFS-fs (dm-0): recovery complete [ 4561.947919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4575.726625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 4576.006550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.008100] Lustre: Skipped 7 previous similar messages [ 4581.353709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4581.364251] Lustre: Skipped 32 previous similar messages [ 4581.576121] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 4581.577197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2866 to 0x2c0000400:2881) [ 4582.679703] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4591.056453] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4592.833460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4597.727397] LustreError: 136665:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4597.741200] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4598.818536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2867 to 0x280000400:2913) [ 4614.106229] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 21:03:33 (1767319413) [ 4619.847479] LustreError: 18722:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4624.863851] LustreError: 18722:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4624.879747] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnecting [ 4624.946975] LustreError: 18317:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 4626.931935] LustreError: 15434:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4632.031150] LustreError: 15434:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4632.045914] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnecting [ 4634.044312] LustreError: 15434:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4639.217383] LustreError: 15434:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4639.230368] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnecting [ 4641.428285] LustreError: 40534:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4646.879166] LustreError: 40534:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4648.938524] LustreError: 22085:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4654.052958] LustreError: 22085:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4654.060651] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnecting [ 4654.069587] Lustre: Skipped 1 previous similar message [ 4663.942523] LustreError: 15433:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4663.947868] LustreError: 15433:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4669.408044] LustreError: 15433:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4669.419828] LustreError: 15433:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4677.088065] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnecting [ 4677.104868] Lustre: Skipped 2 previous similar messages [ 4687.259736] LustreError: 15433:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4687.276662] LustreError: 15433:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4692.447626] LustreError: 15433:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4692.460612] LustreError: 15433:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4702.790446] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 21:05:02 (1767319502) [ 4705.028633] LustreError: 40534:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4715.392479] Lustre: lustre-MDT0000: Export ffff9bb490707000 already connecting from 192.168.203.25@tcp [ 4716.411652] Lustre: lustre-MDT0000: Export ffff9bb490707000 already connecting from 192.168.203.25@tcp [ 4721.531181] Lustre: lustre-MDT0000: Export ffff9bb490707000 already connecting from 192.168.203.25@tcp [ 4726.656994] Lustre: lustre-MDT0000: Export ffff9bb490707000 already connecting from 192.168.203.25@tcp [ 4731.771658] Lustre: lustre-MDT0000: Export ffff9bb490707000 already connecting from 192.168.203.25@tcp [ 4731.784149] Lustre: Skipped 1 previous similar message [ 4742.024830] Lustre: lustre-MDT0000: Export ffff9bb490707000 already connecting from 192.168.203.25@tcp [ 4742.027688] Lustre: Skipped 1 previous similar message [ 4745.079258] LustreError: 40534:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4745.101455] Lustre: 40534:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9bb5a613ce00 x1853163955834880/t0(0) o38->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:0/0 lens 520/416 e 0 to 0 dl 1767319527 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4747.131737] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnecting [ 4747.148458] Lustre: Skipped 3 previous similar messages [ 4747.153747] LustreError: 102716:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4772.740356] Lustre: lustre-MDT0000: Export ffff9bb490707000 already connecting from 192.168.203.25@tcp [ 4787.256893] LustreError: 102716:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4787.261332] Lustre: 102716:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9bb5aa00ca80 x1853163955839232/t0(0) o38->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:0/0 lens 520/416 e 0 to 0 dl 1767319569 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4788.102808] LustreError: 15433:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4813.691506] Lustre: lustre-MDT0000: Export ffff9bb490707000 already connecting from 192.168.203.25@tcp [ 4813.699914] Lustre: Skipped 3 previous similar messages [ 4828.119356] LustreError: 15433:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4828.130594] Lustre: 15433:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9bb5a7d3ea00 x1853163955842432/t0(0) o38->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:0/0 lens 520/416 e 0 to 0 dl 1767319610 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4829.055754] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnecting [ 4829.068813] Lustre: Skipped 1 previous similar message [ 4829.070508] LustreError: 102716:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4869.079676] LustreError: 102716:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4869.092417] Lustre: 102716:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9bb5b3289c00 x1853163955845632/t0(0) o38->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:0/0 lens 520/416 e 0 to 0 dl 1767319651 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4870.011859] LustreError: 102716:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4895.616824] Lustre: lustre-MDT0000: Export ffff9bb490707000 already connecting from 192.168.203.25@tcp [ 4895.626851] Lustre: Skipped 7 previous similar messages [ 4910.087195] LustreError: 102716:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4910.091737] Lustre: 102716:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9bb5b32ba300 x1853163955848832/t0(0) o38->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:0/0 lens 520/416 e 0 to 0 dl 1767319692 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4910.972716] LustreError: 102716:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4935.931608] LustreError: 102716:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 4935.938505] Lustre: 102716:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/5s); client may timeout req@ffff9bb5a7e70380 x1853163955851648/t0(0) o38->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:0/0 lens 520/416 e 0 to 0 dl 1767319733 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4941.193071] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 21:09:00 (1767319740) [ 4949.392327] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4953.728809] Lustre: Failing over lustre-MDT0000 [ 4954.018546] LustreError: 140768:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4954.028255] LustreError: 140768:0:(obd_class.h:479:obd_check_dev()) Skipped 35 previous similar messages [ 4954.238260] Lustre: server umount lustre-MDT0000 complete [ 4955.617891] LustreError: 26879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4955.635870] LustreError: 26879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 196 previous similar messages [ 4971.988550] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4971.991488] LDISKFS-fs (dm-0): recovery complete [ 4971.997778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4972.200437] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4972.220306] LustreError: Skipped 3 previous similar messages [ 4972.429466] Lustre: *** cfs_fail_loc=712, val=0*** [ 4972.436687] LustreError: 18317:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9bb5b32ba680 x1853163989389952/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4972.553052] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4972.553290] Lustre: lustre-MDT0000: Aborting client recovery [ 4972.558816] Lustre: Skipped 2 previous similar messages [ 4972.576157] LustreError: 141558:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4972.588796] Lustre: 141591:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4972.594191] Lustre: 141591:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 4972.611337] Lustre: 141591:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4972.628247] Lustre: 141591:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4972.638174] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4972.645291] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4972.658324] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4972.731248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2867 to 0x280000400:2945) [ 4972.748676] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2866 to 0x2c0000400:2913) [ 4977.467730] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4977.645057] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4994.414671] Lustre: Failing over lustre-MDT0000 [ 4994.748199] Lustre: server umount lustre-MDT0000 complete [ 5014.495122] Lustre: 6993:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319800/real 1767319800] req@ffff9bb5b3800380 x1853163989409536/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767319816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5014.542466] Lustre: 6993:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5023.045515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5024.351140] LustreError: 142887:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 5024.374371] LustreError: 142887:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb5a663ca80 x1853163989415424/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767319826 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5024.409123] LustreError: 142887:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 5024.737979] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5a663ca80 x1853163989419136/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5024.765773] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [ 5025.869105] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5025.872792] Lustre: Skipped 3 previous similar messages [ 5028.107069] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5028.125981] Lustre: Skipped 3 previous similar messages [ 5028.179427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2867 to 0x280000400:2977) [ 5028.182956] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2866 to 0x2c0000400:2945) [ 5030.214712] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5038.886617] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5040.830973] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5049.861591] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 21:10:49 (1767319849) [ 5050.048228] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnecting [ 5050.059105] Lustre: Skipped 3 previous similar messages [ 5058.459858] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 21:10:57 (1767319857) [ 5059.868586] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 5059.870683] LustreError: 15442:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb58a0bfc50 x1853163955913856/t0(0) o700->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:18/0 lens 264/248 e 0 to 0 dl 1767319873 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 5078.533786] Lustre: Failing over lustre-MDT0000 [ 5078.805956] Lustre: server umount lustre-MDT0000 complete [ 5079.521522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5079.534331] LustreError: Skipped 3 previous similar messages [ 5104.238590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.503273] LustreError: 144723:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 5105.523826] LustreError: 144723:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb5a7d38a80 x1853163989458688/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767319907 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5105.563513] LustreError: 144723:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 5105.632580] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5a7d3b480 x1853163989461376/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5110.554825] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5111.328906] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2947 to 0x2c0000400:2977) [ 5111.330956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:2979 to 0x280000400:3009) [ 5118.930385] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5121.452935] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5134.451379] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 21:12:13 (1767319933) [ 5137.536770] Lustre: Failing over lustre-OST0000 [ 5137.686183] Lustre: server umount lustre-OST0000 complete [ 5140.448603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5140.467377] Lustre: Skipped 16 previous similar messages [ 5162.726532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5170.403829] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5178.861364] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5181.428151] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5256.140726] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 21:14:15 (1767320055) [ 5265.262746] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5267.688958] Lustre: Failing over lustre-MDT0000 [ 5267.922828] Lustre: server umount lustre-MDT0000 complete [ 5298.369175] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5298.374438] LDISKFS-fs (dm-0): recovery complete [ 5298.391093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.957159] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d13be81 [ 5310.986243] Lustre: MGC192.168.203.125@tcp: Connection restored to 0@lo (at 0@lo) [ 5310.994426] Lustre: Skipped 18 previous similar messages [ 5311.274404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5311.280709] Lustre: Skipped 4 previous similar messages [ 5316.287105] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5317.218106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3030 to 0x280000400:3073) [ 5317.218376] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:2998 to 0x2c0000400:3041) [ 5390.460711] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 21:16:29 (1767320189) [ 5393.015309] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5393.020100] Lustre: Skipped 2 previous similar messages [ 5405.964529] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 21:16:45 (1767320205) [ 5409.393874] Lustre: Failing over lustre-MDT0000 [ 5409.664433] Lustre: server umount lustre-MDT0000 complete [ 5435.433816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.127613] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5445.134436] LustreError: 150253:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb5a6637480 x1853163956049408/t0(0) o101->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:443/0 lens 328/344 e 0 to 0 dl 1767320298 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 5445.744715] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5500.285065] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [ 5500.309068] Lustre: 150253:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5500.411185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3085 to 0x280000400:3105) [ 5500.419659] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3052 to 0x2c0000400:3073) [ 5505.489821] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5507.547085] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5519.625510] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 21:18:38 (1767320318) [ 5521.977889] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5531.240721] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5533.157068] Lustre: Failing over lustre-MDT0000 [ 5533.484509] Lustre: server umount lustre-MDT0000 complete [ 5558.672838] LustreError: 22085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.710669] LustreError: 22085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 212 previous similar messages [ 5564.856747] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5564.858450] LDISKFS-fs (dm-0): recovery complete [ 5564.867788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5579.417246] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5579.419784] Lustre: Skipped 7 previous similar messages [ 5584.769752] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5584.989511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3107 to 0x280000400:3137) [ 5585.001163] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3052 to 0x2c0000400:3105) [ 5593.012924] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5594.822589] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5603.870200] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 21:20:03 (1767320403) [ 5605.041831] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5615.652043] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5617.627656] Lustre: Failing over lustre-MDT0000 [ 5617.773796] LustreError: 153814:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5617.784048] LustreError: 153814:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 5618.038648] Lustre: server umount lustre-MDT0000 complete [ 5636.069996] Lustre: 6992:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320422/real 1767320422] req@ffff9bb5b3803b80 x1853163989729280/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767320438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5636.107549] Lustre: 6992:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5636.120713] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5636.129405] LustreError: Skipped 5 previous similar messages [ 5649.129294] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5649.135048] LDISKFS-fs (dm-0): recovery complete [ 5649.145800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5662.100040] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5662.106323] Lustre: Skipped 5 previous similar messages [ 5666.544898] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5667.352576] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5667.359378] Lustre: Skipped 5 previous similar messages [ 5667.411168] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3052 to 0x2c0000400:3137) [ 5667.414350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3139 to 0x280000400:3169) [ 5675.001744] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5676.785483] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5685.170350] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 21:21:24 (1767320484) [ 5686.279832] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5688.596712] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5697.945768] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5700.388926] Lustre: Failing over lustre-MDT0000 [ 5700.576669] Lustre: server umount lustre-MDT0000 complete [ 5703.140213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5703.143323] LustreError: Skipped 4 previous similar messages [ 5731.143429] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5731.144978] LDISKFS-fs (dm-0): recovery complete [ 5731.159567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5749.647117] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5749.882064] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3139 to 0x2c0000400:3169) [ 5749.889599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3139 to 0x280000400:3201) [ 5763.251751] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 21:22:42 (1767320562) [ 5765.585206] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5765.590098] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5765.595015] LustreError: 15437:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb5a623ea00 x1853163956118528/t257698037777(0) o35->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:723/0 lens 392/456 e 0 to 0 dl 1767320578 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5768.493108] Lustre: Failing over lustre-MDT0000 [ 5768.997718] Lustre: server umount lustre-MDT0000 complete [ 5770.208745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.229145] Lustre: Skipped 25 previous similar messages [ 5796.048290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5796.833409] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5b33a1500 x1853163989810176/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5801.791569] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5802.568475] Lustre: 15437:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5b987df80 x1853163956118528/t257698037777(0) o35->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:5/0 lens 392/456 e 0 to 0 dl 1767320615 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5802.573216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3139 to 0x280000400:3233) [ 5802.580125] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3171 to 0x2c0000400:3201) [ 5810.248792] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5811.836867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5821.084736] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 21:23:40 (1767320620) [ 5822.251261] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5822.253251] LustreError: 15434:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb5a6638a80 x1853163956134144/t261993005072(0) o36->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:69/0 lens 504/448 e 0 to 0 dl 1767320679 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5832.808317] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5835.069709] Lustre: Failing over lustre-MDT0000 [ 5835.421575] Lustre: server umount lustre-MDT0000 complete [ 5864.320417] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5864.323350] LDISKFS-fs (dm-0): recovery complete [ 5864.338835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.335372] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5884.980301] Lustre: 18722:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb48dd79500 x1853163956134144/t261993005072(0) o36->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:132/0 lens 504/2880 e 0 to 0 dl 1767320742 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5884.995770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3235 to 0x280000400:3265) [ 5884.995992] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3171 to 0x2c0000400:3233) [ 5891.949275] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5893.481791] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5902.197253] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 21:25:01 (1767320701) [ 5903.439817] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5903.442042] LustreError: 40534:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb5a6afc700 x1853163956152192/t266287972368(0) o36->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:150/0 lens 504/448 e 0 to 0 dl 1767320760 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5905.330960] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5912.847521] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5914.745359] Lustre: Failing over lustre-MDT0000 [ 5915.111610] Lustre: server umount lustre-MDT0000 complete [ 5944.369580] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5944.371778] LDISKFS-fs (dm-0): recovery complete [ 5944.381910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.607578] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d13f7ae [ 5956.631518] Lustre: Skipped 2 previous similar messages [ 5956.641942] Lustre: MGC192.168.203.125@tcp: Connection restored to 0@lo (at 0@lo) [ 5956.657833] Lustre: Skipped 32 previous similar messages [ 5956.932488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5956.940545] Lustre: Skipped 6 previous similar messages [ 5961.826930] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5962.248056] Lustre: 15437:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5863f6300 x1853163956152448/t266287972369(0) o35->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:165/0 lens 392/456 e 0 to 0 dl 1767320775 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5962.264862] Lustre: 15437:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5962.275060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3235 to 0x280000400:3297) [ 5962.275939] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3235 to 0x2c0000400:3265) [ 5973.247523] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 21:26:12 (1767320772) [ 5974.387112] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5974.391383] Lustre: Skipped 1 previous similar message [ 5974.395951] LustreError: 40534:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb5bd77c380 x1853163956167552/t270582939664(0) o36->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:221/0 lens 504/448 e 0 to 0 dl 1767320831 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5974.415990] LustreError: 40534:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 5976.356530] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5985.339684] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5987.623595] Lustre: Failing over lustre-MDT0000 [ 5987.888265] Lustre: server umount lustre-MDT0000 complete [ 6015.894483] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6015.898034] LDISKFS-fs (dm-0): recovery complete [ 6015.908251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6023.846286] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6025.277527] Lustre: 15433:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5bd608e00 x1853163956167552/t270582939664(0) o36->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:272/0 lens 504/2880 e 0 to 0 dl 1767320882 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 6025.309221] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3235 to 0x2c0000400:3297) [ 6025.309863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3299 to 0x280000400:3329) [ 6035.684811] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 21:27:15 (1767320835) [ 6036.727156] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 6038.807754] Lustre: *** cfs_fail_loc=13b, val=315*** [ 6038.812668] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 6038.830058] LustreError: 15437:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb585d9a300 x1853163956182016/t274877906960(0) o35->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:241/0 lens 392/456 e 0 to 0 dl 1767320851 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 6048.696049] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6050.893985] Lustre: Failing over lustre-MDT0000 [ 6051.144247] Lustre: server umount lustre-MDT0000 complete [ 6080.084857] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6080.091106] LDISKFS-fs (dm-0): recovery complete [ 6080.109142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6087.494562] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6087.725059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3331 to 0x280000400:3361) [ 6087.726137] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3235 to 0x2c0000400:3329) [ 6087.743528] Lustre: 15437:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5b3320000 x1853163956182016/t274877906960(0) o35->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:290/0 lens 392/456 e 0 to 0 dl 1767320900 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 6100.019424] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 21:28:19 (1767320899) [ 6101.170176] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 6101.182866] LustreError: 15433:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb5863be050 x1853163956195200/t279172874255(0) o101->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:348/0 lens 664/608 e 0 to 0 dl 1767320958 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 6161.798227] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnecting [ 6161.807120] Lustre: Skipped 1 previous similar message [ 6161.828923] Lustre: 15433:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5890a4380 x1853163956195200/t279172874255(0) o101->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:408/0 lens 664/3488 e 0 to 0 dl 1767321018 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 6168.411089] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 21:29:28 (1767320968) [ 6175.719580] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6177.739748] Lustre: Failing over lustre-MDT0000 [ 6178.089081] Lustre: server umount lustre-MDT0000 complete [ 6179.807722] LustreError: 26879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.819218] LustreError: 26879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 324 previous similar messages [ 6206.185142] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 6206.192909] LDISKFS-fs (dm-0): recovery complete [ 6206.223976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6221.792367] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5a7e0d880 x1853163990026624/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6221.821245] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 40 previous similar messages [ 6222.209429] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6222.218852] Lustre: Skipped 7 previous similar messages [ 6227.092176] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6227.548596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3331 to 0x2c0000400:3361) [ 6227.551054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3331 to 0x280000400:3393) [ 6236.131107] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6238.686592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6257.839574] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 21:30:57 (1767321057) [ 6266.290533] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6268.350451] Lustre: Failing over lustre-MDT0000 [ 6268.385808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6268.501668] LustreError: 170737:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6268.510738] LustreError: 170737:0:(obd_class.h:479:obd_check_dev()) Skipped 63 previous similar messages [ 6268.741179] Lustre: server umount lustre-MDT0000 complete [ 6288.868253] Lustre: 6993:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767321075/real 1767321075] req@ffff9bb5a7d3ca80 x1853163990059776/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767321091 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6288.903050] Lustre: 6993:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 6288.920379] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6288.943636] LustreError: Skipped 7 previous similar messages [ 6298.229686] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6298.231449] LDISKFS-fs (dm-0): recovery complete [ 6298.235805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6302.084315] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6302.094291] Lustre: Skipped 7 previous similar messages [ 6303.589852] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6304.835192] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6304.843617] Lustre: Skipped 7 previous similar messages [ 6304.902361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3331 to 0x280000400:3425) [ 6304.903373] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:3363 to 0x2c0000400:3393) [ 6311.392277] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6312.985677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6318.489575] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 6330.774581] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 21:32:10 (1767321130) [ 6374.890776] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6376.883157] Lustre: Failing over lustre-MDT0000 [ 6377.281104] Lustre: server umount lustre-MDT0000 complete [ 6381.537766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6381.538942] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.553682] LustreError: Skipped 5 previous similar messages [ 6381.585162] Lustre: Skipped 27 previous similar messages [ 6404.043772] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6404.046701] LDISKFS-fs (dm-0): recovery complete [ 6404.054378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.309589] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6413.165407] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4644 to 0x2c0000400:4673) [ 6413.168938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4676 to 0x280000400:4705) [ 6420.067580] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6421.876265] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6472.398866] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 21:34:32 (1767321272) [ 6482.129919] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6484.517109] Lustre: Failing over lustre-MDT0000 [ 6484.863210] Lustre: server umount lustre-MDT0000 complete [ 6512.362263] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6512.364756] LDISKFS-fs (dm-0): recovery complete [ 6512.370461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.550420] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6521.441037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4707 to 0x280000400:4737) [ 6521.444242] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4644 to 0x2c0000400:4705) [ 6528.161173] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6529.931421] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6540.207256] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 6541.603884] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 6548.091429] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 21:35:47 (1767321347) [ 6556.826631] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 6618.990837] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6619.000948] Lustre: Skipped 1 previous similar message [ 6619.014027] LustreError: 18722:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb5bd60ca80 x1853163959004800/t296352743435(0) o36->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:111/0 lens 66040/440 e 0 to 0 dl 1767321476 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 6678.921024] Lustre: 15433:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb582d23050 x1853163959004800/t296352743435(0) o36->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:171/0 lens 66040/440 e 0 to 0 dl 1767321536 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 6689.824722] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 6691.466760] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 21:38:11 (1767321491) [ 6705.225334] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6707.858273] Lustre: Failing over lustre-MDT0000 [ 6708.396822] Lustre: server umount lustre-MDT0000 complete [ 6736.251166] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6736.253868] LDISKFS-fs (dm-0): recovery complete [ 6736.264430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.358217] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d184c40 [ 6736.364552] Lustre: Skipped 1 previous similar message [ 6736.367713] Lustre: MGC192.168.203.125@tcp: Connection restored to 0@lo (at 0@lo) [ 6736.376469] Lustre: Skipped 29 previous similar messages [ 6736.538515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6736.545265] Lustre: Skipped 6 previous similar messages [ 6740.578113] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6742.973440] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4806 to 0x2c0000400:4833) [ 6742.974961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4839 to 0x280000400:4865) [ 6749.069890] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6751.074289] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6761.523434] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 21:39:21 (1767321561) [ 6782.349578] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6794.217141] Lustre: Failing over lustre-OST0000 [ 6794.313523] Lustre: server umount lustre-OST0000 complete [ 6794.734841] LustreError: 57644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6794.744978] LustreError: 57644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 183 previous similar messages [ 6820.041947] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6820.045937] LDISKFS-fs (dm-2): recovery complete [ 6820.052940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6826.651551] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6841.344835] Lustre: Failing over lustre-OST0000 [ 6841.348723] LustreError: 182802:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 6841.356437] Lustre: 182253:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6841.362902] Lustre: 182253:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 6841.368226] Lustre: 182253:0:(ldlm_lib.c:1897:abort_req_replay_queue()) @@@ aborted: req@ffff9bb584057800 x1853163991006208/t0(17179870646) o6->lustre-MDT0001-mdtlov_UUID@0@lo:294/0 lens 544/0 e 2 to 0 dl 1767321659 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'osp-syn-0-1.0' uid:0 gid:0 projid:4294967295 [ 6841.382044] LustreError: 182253:0:(ofd_obd.c:1293:ofd_iocontrol()) lustre-OST0000: iocontrol from 'tgt_recover_0' cmd=c00866c1 _IOWR('f', 193, 8) unrecognized: rc = -25 [ 6841.387501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6841.397463] Lustre: Skipped 3 previous similar messages [ 6841.492570] Lustre: server umount lustre-OST0000 complete [ 6863.541869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6863.926833] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6863.941781] Lustre: Skipped 5 previous similar messages [ 6870.256859] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6874.312096] LustreError: 6990:0:(client.c:3418:ptlrpc_replay_interpret()) @@@ status 0, old was -19 req@ffff9bb582cd8000 x1853163991006208/t17179870646(17179870646) o6->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 544/432 e 3 to 0 dl 1767321696 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'osp-syn-0-1.0' uid:0 gid:0 projid:4294967295 [ 6879.413618] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6881.245512] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6921.951663] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 21:42:01 (1767321721) [ 6924.733928] Lustre: Failing over lustre-MDT0000 [ 6924.901298] LustreError: 184377:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6924.913133] LustreError: 184377:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 6925.041841] Lustre: server umount lustre-MDT0000 complete [ 6943.711121] Lustre: 6993:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767321729/real 1767321729] req@ffff9bb5890cb800 x1853163991174656/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767321745 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6943.731309] Lustre: 6993:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6943.738234] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6943.761694] LustreError: Skipped 3 previous similar messages [ 6948.235033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6953.747954] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6953.757208] Lustre: Skipped 5 previous similar messages [ 6957.962923] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6958.571931] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6958.586636] Lustre: Skipped 5 previous similar messages [ 6958.652038] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4806 to 0x2c0000400:4865) [ 6958.652996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4839 to 0x280000400:4897) [ 6972.785756] Lustre: Failing over lustre-MDT0000 [ 6973.030882] Lustre: server umount lustre-MDT0000 complete [ 6995.880908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7006.145285] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7006.303957] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4806 to 0x2c0000400:4897) [ 7006.306218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4839 to 0x280000400:4929) [ 7014.624717] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7016.554375] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7025.068883] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 21:43:44 (1767321824) [ 7039.256103] Lustre: Failing over lustre-OST0000 [ 7039.634978] Lustre: server umount lustre-OST0000 complete [ 7042.018442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.050914] Lustre: Skipped 20 previous similar messages [ 7059.766135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7065.253400] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7072.361672] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7073.866878] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7083.238981] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 21:44:42 (1767321882) [ 7087.557345] Lustre: Failing over lustre-MDT0000 [ 7087.981538] Lustre: server umount lustre-MDT0000 complete [ 7088.096702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7088.110408] LustreError: Skipped 12 previous similar messages [ 7106.416672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.481872] Lustre: *** cfs_fail_loc=605, val=0*** [ 7106.484463] LustreError: 189990:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc07c7ab0 failed: rc = -95 [ 7106.490195] LustreError: 189990:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 7106.494463] LustreError: 189990:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 7106.498635] LustreError: 189990:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 7106.500359] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 7106.501917] LustreError: 189990:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 7106.509278] Lustre: server umount lustre-MDT0000 complete [ 7106.512316] LustreError: 189990:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 7128.153063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7131.616529] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5a6636d80 x1853163991278464/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7131.661117] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 9 previous similar messages [ 7136.622371] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7137.328927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4931 to 0x280000400:4961) [ 7137.332382] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4899 to 0x2c0000400:4929) [ 7145.549582] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 21:45:44 (1767321944) [ 7152.928361] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7156.891175] Lustre: Failing over lustre-MDT0000 [ 7159.112292] Lustre: server umount lustre-MDT0000 complete [ 7189.104975] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7189.108201] LDISKFS-fs (dm-0): recovery complete [ 7189.115938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.657803] Lustre: *** cfs_fail_loc=707, val=0*** [ 7204.117247] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7260.555827] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7260.638272] Lustre: 193114:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 7260.662642] Lustre: 193114:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 7261.137038] Lustre: 193114:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 7261.154577] Lustre: 193114:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 50 previous similar messages [ 7261.232962] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:4942 to 0x2c0000400:4961) [ 7261.236452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:4975 to 0x280000400:4993) [ 7266.787754] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7268.671108] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7279.837651] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 21:47:59 (1767322079) [ 7310.372394] LustreError: 15435:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9bb586e2ca80 x1853163959908224/t0(0) o101->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:3/0 lens 664/0 e 0 to 0 dl 1767322123 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7310.399968] LustreError: 15435:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 7321.471132] LustreError: 15435:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7321.484591] LustreError: 53212:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9bb5812e9500 x1853163959909248/t0(0) o35->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:14/0 lens 392/0 e 0 to 0 dl 1767322134 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7323.506397] LustreError: 15434:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9bb5b3801f80 x1853163959915136/t0(0) o101->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:55/0 lens 576/0 e 0 to 0 dl 1767322175 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 7323.542559] LustreError: 15434:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 7327.206389] LustreError: 57646:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9bb5a5c79500 x1853163991375616/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:20/0 lens 544/0 e 0 to 0 dl 1767322140 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 7327.232353] LustreError: 57646:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 20 previous similar messages [ 7332.835341] LustreError: 57643:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9bb482794e00 x1853163991378560/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:25/0 lens 544/0 e 0 to 0 dl 1767322145 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 7332.855247] LustreError: 57643:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 7 previous similar messages [ 7340.912214] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 21:49:00 (1767322140) [ 7371.178637] LustreError: 56308:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 7382.247146] LustreError: 56308:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 7391.168960] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 21:49:51 (1767322191) [ 7419.919214] LustreError: 102716:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9bb5aec92d80 x1853163959933440/t0(0) o101->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:113/0 lens 576/0 e 0 to 0 dl 1767322233 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7419.944606] LustreError: 102716:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 7424.951711] LustreError: 102716:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7437.792908] LustreError: 102716:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7437.854333] LustreError: 102716:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9bb5a7e73b80 x1853163959956096/t0(0) o101->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:170/0 lens 664/0 e 0 to 0 dl 1767322290 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7437.910421] LustreError: 102716:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 120 previous similar messages [ 7457.889532] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 21:50:57 (1767322257) [ 7561.389787] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 21:52:40 (1767322360) [ 7592.391682] LustreError: 22085:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9bb4905f4e00 x1853163960014848/t0(0) o101->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:285/0 lens 576/0 e 0 to 0 dl 1767322405 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7592.401459] LustreError: 22085:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 7592.403863] LustreError: 22085:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 7592.407529] LustreError: 22085:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 7592.823371] LustreError: 22085:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7608.726237] LustreError: 15436:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 7608.731557] LustreError: 15436:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 7609.159743] LustreError: 15436:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7609.174531] LustreError: 15436:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 7642.204926] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 21:54:01 (1767322441) [ 7675.581106] Lustre: DEBUG MARKER: phase 2 [ 7685.867689] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 21:54:45 (1767322485) [ 7769.162439] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 21:56:08 (1767322568) [ 7770.650173] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 7772.117680] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 21:56:11 (1767322571) [ 7776.689566] Lustre: DEBUG MARKER: Started rundbench load pid=141086 ... [ 7785.501581] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7787.987040] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 7789.781594] Lustre: Failing over lustre-MDT0000 [ 7789.914428] LustreError: 200078:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7789.919513] LustreError: 200078:0:(obd_class.h:479:obd_check_dev()) Skipped 35 previous similar messages [ 7790.014938] Lustre: server umount lustre-MDT0000 complete [ 7791.086405] LustreError: 18722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.110175] LustreError: 18722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 188 previous similar messages [ 7792.619202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.633868] Lustre: Skipped 10 previous similar messages [ 7808.993978] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767322594/real 1767322594] req@ffff9bb48bc7bb80 x1853163991648640/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767322610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7809.023387] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 7809.030759] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7809.044838] LustreError: Skipped 3 previous similar messages [ 7815.537267] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 7815.548693] LDISKFS-fs (dm-0): recovery complete [ 7815.568068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.471553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7819.489462] Lustre: Skipped 7 previous similar messages [ 7819.541913] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7819.551775] Lustre: Skipped 5 previous similar messages [ 7820.155647] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7820.158842] Lustre: Skipped 4 previous similar messages [ 7823.699639] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7824.882899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7824.886823] Lustre: Skipped 24 previous similar messages [ 7824.948580] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7824.958843] Lustre: Skipped 4 previous similar messages [ 7824.996375] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:5011 to 0x2c0000400:5057) [ 7824.996418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5046 to 0x280000400:5089) [ 7831.185721] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7833.074214] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7843.743803] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7846.868433] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 7848.968274] Lustre: Failing over lustre-MDT0001 [ 7849.016920] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.25@tcp (stopping) [ 7849.342518] Lustre: server umount lustre-MDT0001 complete [ 7874.403161] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7874.407414] LDISKFS-fs (dm-1): recovery complete [ 7874.424620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7878.641249] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7881.600097] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:572 to 0x280000401:609) [ 7881.600163] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:554 to 0x2c0000401:577) [ 7885.967158] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7887.419856] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7896.424290] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7898.965361] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 7900.841043] Lustre: Failing over lustre-MDT0000 [ 7901.178816] Lustre: server umount lustre-MDT0000 complete [ 7905.248444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7905.252704] LustreError: Skipped 1 previous similar message [ 7925.522405] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7925.524575] LDISKFS-fs (dm-0): recovery complete [ 7925.533167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7931.359528] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5883b6680 x1853163992046336/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7931.380950] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [ 7935.130077] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7937.063584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5046 to 0x280000400:5121) [ 7937.067230] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:5011 to 0x2c0000400:5089) [ 7941.510530] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7943.264726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7949.647715] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 21:59:09 (1767322749) [ 8075.870734] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8087.218346] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 8088.390308] Lustre: Failing over lustre-MDT0001 [ 8088.398146] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.25@tcp (stopping) [ 8088.627147] Lustre: server umount lustre-MDT0001 complete [ 8109.085987] LDISKFS-fs (dm-1): 7 truncates cleaned up [ 8109.089398] LDISKFS-fs (dm-1): recovery complete [ 8109.100510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8112.426315] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8118.556779] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:1014 to 0x280000401:1057) [ 8118.557497] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:982 to 0x2c0000401:1025) [ 8121.236283] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8122.454908] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8249.815628] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8261.321995] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 8262.622704] Lustre: Failing over lustre-MDT0000 [ 8262.646299] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.25@tcp (stopping) [ 8262.982857] Lustre: server umount lustre-MDT0000 complete [ 8286.201189] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8286.203985] LDISKFS-fs (dm-0): recovery complete [ 8286.210714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.777041] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d262b77 [ 8293.168365] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8298.907658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:5835 to 0x280000400:5857) [ 8298.908653] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:5803 to 0x2c0000400:5825) [ 8301.721714] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8302.782647] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8364.776870] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 22:06:04 (1767323164) [ 8491.952085] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8503.597623] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 8504.885481] Lustre: Failing over lustre-MDT0001 [ 8504.964206] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.25@tcp (stopping) [ 8505.066202] LustreError: 210222:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8505.072132] LustreError: 210222:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 8505.185536] Lustre: server umount lustre-MDT0001 complete [ 8505.313515] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8505.317863] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8505.320052] LustreError: Skipped 2 previous similar messages [ 8505.324363] LustreError: 15438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8505.340021] Lustre: Skipped 17 previous similar messages [ 8505.358872] LustreError: 15438:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 135 previous similar messages [ 8526.643314] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8526.646399] LDISKFS-fs (dm-1): recovery complete [ 8526.658798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8526.861584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8526.866348] Lustre: Skipped 4 previous similar messages [ 8526.895680] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8526.899785] Lustre: Skipped 4 previous similar messages [ 8528.419449] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8528.427497] Lustre: Skipped 4 previous similar messages [ 8530.014247] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8531.942920] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 8531.948299] Lustre: Skipped 18 previous similar messages [ 8533.248295] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8533.257663] Lustre: Skipped 4 previous similar messages [ 8533.288971] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:1562 to 0x2c0000401:1601) [ 8533.289380] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:1594 to 0x280000401:1633) [ 8536.330474] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8537.763647] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8665.290785] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8676.777884] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 8678.047142] Lustre: Failing over lustre-MDT0000 [ 8678.074860] LustreError: 18342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323480 with bad export cookie 5068692619638114410 [ 8678.088659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8684.236193] Lustre: server umount lustre-MDT0000 complete [ 8701.919825] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767323487/real 1767323487] req@ffff9bb5a62d6300 x1853164000251264/t0(0) o400->MGC192.168.203.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767323503 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8701.940668] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 8701.950970] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8701.957410] LustreError: Skipped 2 previous similar messages [ 8706.316779] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8706.320409] LDISKFS-fs (dm-0): recovery complete [ 8706.329826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8712.160329] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb5883b4a80 x1853164000260352/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8712.193115] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 8714.410001] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8719.227046] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:6175 to 0x2c0000400:6209) [ 8719.227779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6208 to 0x280000400:6241) [ 8721.568325] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8722.628329] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8727.498161] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 22:12:07 (1767323527) [ 8852.400672] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8863.665456] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 8864.870931] Lustre: Failing over lustre-MDT0000 [ 8864.897431] LustreError: 6993:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9bb5adf9f100 x1853164001621376/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 8864.935476] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.25@tcp (stopping) [ 8864.938636] Lustre: Skipped 5 previous similar messages [ 8865.063565] Lustre: server umount lustre-MDT0000 complete [ 8884.847400] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8884.852253] LDISKFS-fs (dm-0): recovery complete [ 8884.860140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.035262] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8900.963868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:6969 to 0x280000400:7009) [ 8900.963935] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:6936 to 0x2c0000400:6977) [ 8903.093776] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8904.158033] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9029.358383] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9040.373332] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 9041.202745] Lustre: Failing over lustre-MDT0001 [ 9041.221130] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9041.394692] Lustre: server umount lustre-MDT0001 complete [ 9057.729675] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9057.731561] LDISKFS-fs (dm-1): recovery complete [ 9057.736251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9059.368568] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9063.434679] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:1784 to 0x280000401:1825) [ 9063.434693] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:1753 to 0x2c0000401:1793) [ 9065.365543] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9066.228746] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9071.284439] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 22:17:51 (1767323871) [ 9078.642845] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9080.449301] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 9081.426753] Lustre: Failing over lustre-OST0000 [ 9081.474985] Lustre: server umount lustre-OST0000 complete [ 9098.844836] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9098.846498] LDISKFS-fs (dm-2): recovery complete [ 9098.853608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9101.597298] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9105.758331] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9106.621229] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9116.634205] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9118.476880] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 9119.474871] Lustre: Failing over lustre-OST0000 [ 9119.501168] LustreError: 220217:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 9119.508461] LustreError: 220217:0:(obd_class.h:479:obd_check_dev()) Skipped 35 previous similar messages [ 9119.549782] Lustre: server umount lustre-OST0000 complete [ 9119.713708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9119.714404] LustreError: 57648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9119.719725] Lustre: Skipped 14 previous similar messages [ 9119.731655] LustreError: 57648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 109 previous similar messages [ 9136.431061] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9136.433297] LDISKFS-fs (dm-2): recovery complete [ 9136.438265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9136.510055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9136.513711] Lustre: Skipped 4 previous similar messages [ 9136.520266] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 9136.523821] Lustre: Skipped 4 previous similar messages [ 9137.934810] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9137.938919] Lustre: Skipped 4 previous similar messages [ 9138.230681] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9138.230732] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9138.235843] Lustre: Skipped 4 previous similar messages [ 9138.238441] Lustre: Skipped 15 previous similar messages [ 9139.215238] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9143.360966] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9144.264990] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9150.550774] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 22:19:11 (1767323951) [ 9274.668391] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9277.848581] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9288.984375] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 9289.973753] Lustre: Failing over lustre-MDT0000 [ 9290.003887] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.25@tcp (stopping) [ 9290.206350] Lustre: server umount lustre-MDT0000 complete [ 9292.131054] LustreError: 15426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324094 with bad export cookie 5068692619638809979 [ 9292.131897] Lustre: Failing over lustre-MDT0001 [ 9292.138151] LustreError: 15426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9292.408828] Lustre: server umount lustre-MDT0001 complete [ 9310.182422] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 9310.184472] LDISKFS-fs (dm-0): recovery complete [ 9310.192050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9311.543759] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 9311.545492] LDISKFS-fs (dm-1): recovery complete [ 9311.550816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9311.711339] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767324097/real 1767324097] req@ffff9bb48a46f480 x1853164008175360/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767324113 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9311.721298] Lustre: 6991:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 9317.858644] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb48a46f800 x1853164008177920/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9317.869492] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 9318.095843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 9318.099757] LustreError: Skipped 3 previous similar messages [ 9319.897838] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9319.990239] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9322.347228] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:1784 to 0x280000401:1857) [ 9322.349592] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:1753 to 0x2c0000401:1825) [ 9334.779486] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7912 to 0x2c0000400:7937) [ 9334.784747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:7945 to 0x280000400:7969) [ 9336.917324] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9337.726869] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9338.441220] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9342.389433] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 22:22:22 (1767324142) [ 9345.736100] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9346.932525] Lustre: Failing over lustre-MDT0000 [ 9347.115908] Lustre: server umount lustre-MDT0000 complete [ 9365.153603] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 9365.155247] LDISKFS-fs (dm-0): recovery complete [ 9365.161206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.232659] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9365.237826] LustreError: Skipped 2 previous similar messages [ 9365.890219] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 9367.136082] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9381.254615] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 9381.369360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:7945 to 0x280000400:8001) [ 9381.369371] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7939 to 0x2c0000400:7969) [ 9383.541780] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9384.444295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9388.853302] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 22:23:09 (1767324189) [ 9392.346473] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9393.664526] Lustre: Failing over lustre-MDT0000 [ 9393.865916] Lustre: server umount lustre-MDT0000 complete [ 9412.372376] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9412.374163] LDISKFS-fs (dm-0): recovery complete [ 9412.380525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9412.998227] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 9413.000902] LustreError: 225253:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb5b3a3aa00 x1853163988307328/t339302426104(339302426104) o101->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:596/0 lens 520/664 e 0 to 0 dl 1767324226 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 9414.181564] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9428.360093] Lustre: lustre-MDT0000: Client b170ef68-e829-4c28-bf9f-1c545bc4c72c (at 192.168.203.25@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 9428.371527] Lustre: 225253:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5aa1eb800 x1853163988307328/t339302426104(339302426104) o101->b170ef68-e829-4c28-bf9f-1c545bc4c72c@192.168.203.25@tcp:611/0 lens 520/3488 e 0 to 0 dl 1767324241 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 9428.425151] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8001) [ 9428.425298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:7945 to 0x280000400:8033) [ 9430.205583] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9430.900475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9434.691952] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 22:23:55 (1767324235) [ 9436.015090] Lustre: Failing over lustre-OST0000 [ 9436.067065] Lustre: server umount lustre-OST0000 complete [ 9437.516207] Lustre: Failing over lustre-MDT0000 [ 9437.602469] Lustre: server umount lustre-MDT0000 complete [ 9454.133154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.922993] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9456.817469] Lustre: lustre-MDT0000: Denying connection for new client 7755806c-873d-4fe3-9454-81b9c5460b7f (at 192.168.203.25@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 9456.825625] Lustre: Skipped 12 previous similar messages [ 9459.699336] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8033) [ 9465.045448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9466.494636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:7945 to 0x280000400:8065) [ 9467.866362] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9473.163856] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 22:24:33 (1767324273) [ 9473.621730] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 9473.623539] LustreError: 225333:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb58a340a80 x1853164009031296/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:656/0 lens 1312/4320 e 0 to 0 dl 1767324286 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 9476.796305] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9477.590457] Lustre: Failing over lustre-MDT0000 [ 9477.805756] Lustre: server umount lustre-MDT0000 complete [ 9495.715545] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9495.717410] LDISKFS-fs (dm-0): recovery complete [ 9495.721556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9497.567665] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9501.197906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8097) [ 9501.198052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8065) [ 9502.721619] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9503.386402] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9508.192092] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 22:25:08 (1767324308) [ 9508.708326] LustreError: 224516:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb5b3a3bb80 x1853164009060224/t360777252876(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:691/0 lens 2520/4320 e 0 to 0 dl 1767324321 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 9512.189719] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9515.350888] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9516.100654] Lustre: Failing over lustre-MDT0001 [ 9516.105112] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.25@tcp (stopping) [ 9522.245964] Lustre: server umount lustre-MDT0001 complete [ 9538.771128] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9538.773193] LDISKFS-fs (dm-1): recovery complete [ 9538.779510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9540.681646] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9544.181646] Lustre: 232972:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb4833df100 x1853163991951104/t25769805134(0) o36->7755806c-873d-4fe3-9454-81b9c5460b7f@192.168.203.25@tcp:753/0 lens 560/2880 e 0 to 0 dl 1767324383 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9544.184368] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:1836 to 0x2c0000401:1857) [ 9544.184368] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:1868 to 0x280000401:1889) [ 9545.804234] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9546.618208] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9551.481092] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 22:25:51 (1767324351) [ 9551.983725] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 9551.985805] Lustre: Skipped 1 previous similar message [ 9555.488184] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9559.371838] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9560.231105] Lustre: Failing over lustre-MDT0000 [ 9560.414062] Lustre: server umount lustre-MDT0000 complete [ 9578.750303] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9578.751861] LDISKFS-fs (dm-0): recovery complete [ 9578.756317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9580.633971] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9584.107820] Lustre: 238749:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 9584.113789] Lustre: 238749:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 9584.190576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8129) [ 9584.190664] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8097) [ 9585.761885] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9586.536547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9588.814519] Lustre: Failing over lustre-MDT0001 [ 9588.928053] Lustre: server umount lustre-MDT0001 complete [ 9606.089990] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9606.091887] LDISKFS-fs (dm-1): recovery complete [ 9606.103600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9608.009096] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9611.770165] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:1868 to 0x2c0000401:1889) [ 9611.770171] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:1900 to 0x280000401:1921) [ 9613.535451] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9614.395609] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9618.999344] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 22:26:59 (1767324419) [ 9619.473298] LustreError: 224516:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb5acb78380 x1853164009146112/t365072220176(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:47/0 lens 2520/4320 e 0 to 0 dl 1767324432 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 9619.486019] LustreError: 224516:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 9625.455424] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9628.636872] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9629.423336] Lustre: Failing over lustre-MDT0000 [ 9629.611964] Lustre: server umount lustre-MDT0000 complete [ 9631.165081] LustreError: 15425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324433 with bad export cookie 5068692619639821472 [ 9631.166491] Lustre: Failing over lustre-MDT0001 [ 9631.170676] LustreError: 15425:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9631.174903] LustreError: 241923:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0xa:0x0].0xf7117594 (ffff9bb58abb6e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9636.832351] LustreError: 241923:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9bb481d28a80 x1853164009156992/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 9636.841619] LustreError: 241923:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 9636.845400] LustreError: 241923:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 9636.961429] Lustre: server umount lustre-MDT0001 complete [ 9654.419801] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9654.421246] LDISKFS-fs (dm-0): recovery complete [ 9654.425259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9655.609885] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9655.612021] LDISKFS-fs (dm-1): recovery complete [ 9655.618126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9656.543143] LustreError: 243284:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9656.546496] LustreError: 243284:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb481d28e00 x1853164009157120/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767324458 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9656.554697] LustreError: 243284:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9658.554068] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9658.697246] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9661.172160] Lustre: 243471:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5890a3100 x1853163992021760/t34359738438(0) o36->7755806c-873d-4fe3-9454-81b9c5460b7f@192.168.203.25@tcp:114/0 lens 560/2880 e 0 to 0 dl 1767324499 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9661.175582] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:1932 to 0x280000401:1953) [ 9661.176244] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:1900 to 0x2c0000401:1921) [ 9673.633175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8161) [ 9673.633411] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8129) [ 9675.334676] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9675.964266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9676.629606] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9680.621921] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 22:28:01 (1767324481) [ 9681.058546] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 9681.060694] Lustre: Skipped 1 previous similar message [ 9687.100090] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9687.907820] Lustre: Failing over lustre-MDT0000 [ 9688.059798] Lustre: server umount lustre-MDT0000 complete [ 9705.405992] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9705.408090] LDISKFS-fs (dm-0): recovery complete [ 9705.411778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.296429] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9710.598099] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8161) [ 9710.598251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8193) [ 9711.932445] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9712.509309] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9736.580957] Lustre: lustre-MDT0001: Client 7755806c-873d-4fe3-9454-81b9c5460b7f (at 192.168.203.25@tcp) reconnecting [ 9736.584695] Lustre: Skipped 2 previous similar messages [ 9736.591173] Lustre: 244219:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5b3a39f80 x1853163992055808/t38654705734(0) o36->7755806c-873d-4fe3-9454-81b9c5460b7f@192.168.203.25@tcp:203/0 lens 560/2880 e 0 to 0 dl 1767324588 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9740.611286] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 22:29:00 (1767324540) [ 9744.429185] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9745.206587] Lustre: Failing over lustre-MDT0001 [ 9745.271614] LustreError: 247344:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9745.274802] LustreError: 247344:0:(obd_class.h:479:obd_check_dev()) Skipped 103 previous similar messages [ 9745.295736] Lustre: server umount lustre-MDT0001 complete [ 9745.787087] LustreError: 243472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9745.794376] LustreError: 243472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 157 previous similar messages [ 9746.400117] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9746.405646] Lustre: Skipped 41 previous similar messages [ 9761.309464] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9761.311192] LDISKFS-fs (dm-1): recovery complete [ 9761.317086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9761.429220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9761.433495] Lustre: Skipped 13 previous similar messages [ 9761.442420] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 9761.445552] Lustre: Skipped 13 previous similar messages [ 9762.695084] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9762.697757] Lustre: Skipped 13 previous similar messages [ 9762.956351] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9766.881606] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 9766.884217] Lustre: Skipped 44 previous similar messages [ 9766.890896] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9766.894825] Lustre: Skipped 13 previous similar messages [ 9766.913645] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:1974 to 0x280000401:2017) [ 9766.913664] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:1942 to 0x2c0000401:1985) [ 9768.399448] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9769.095431] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9773.831787] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 22:29:34 (1767324574) [ 9774.278517] LustreError: 243472:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb48502bb80 x1853163992120960/t42949673030(0) o36->7755806c-873d-4fe3-9454-81b9c5460b7f@192.168.203.25@tcp:202/0 lens 560/448 e 0 to 0 dl 1767324587 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9774.291053] LustreError: 243472:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 2 previous similar messages [ 9780.527869] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9783.754990] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9784.459936] Lustre: Failing over lustre-MDT0000 [ 9784.704578] Lustre: server umount lustre-MDT0000 complete [ 9790.844148] Lustre: 244219:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5aa1e8e00 x1853163992120960/t42949673030(0) o36->7755806c-873d-4fe3-9454-81b9c5460b7f@192.168.203.25@tcp:218/0 lens 560/2880 e 0 to 0 dl 1767324603 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9790.854037] Lustre: 244219:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 9802.413851] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9802.415318] LDISKFS-fs (dm-0): recovery complete [ 9802.419842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9804.243318] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9807.887327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8193) [ 9807.887463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8225) [ 9809.372108] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9810.118743] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9812.393545] Lustre: Failing over lustre-MDT0001 [ 9812.478606] Lustre: server umount lustre-MDT0001 complete [ 9829.283875] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9829.285829] LDISKFS-fs (dm-1): recovery complete [ 9829.292549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9830.981383] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9834.485214] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:1996 to 0x2c0000401:2017) [ 9834.485269] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:2028 to 0x280000401:2049) [ 9835.893184] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9836.581312] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9840.702522] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 22:30:41 (1767324641) [ 9846.948271] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9849.871199] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9850.656379] Lustre: Failing over lustre-MDT0000 [ 9850.858067] Lustre: server umount lustre-MDT0000 complete [ 9852.211809] LustreError: 15424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324654 with bad export cookie 5068692619639835227 [ 9852.213099] Lustre: Failing over lustre-MDT0001 [ 9852.215776] LustreError: 15424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9852.317402] Lustre: server umount lustre-MDT0001 complete [ 9869.786476] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9869.788035] LDISKFS-fs (dm-0): recovery complete [ 9869.791912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.871387] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9870.872899] LDISKFS-fs (dm-1): recovery complete [ 9870.877965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9878.797825] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9878.854294] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9881.958074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8257) [ 9881.958082] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8225) [ 9883.023523] Lustre: 256135:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb5bcf28700 x1853163992159488/t47244640326(0) o36->7755806c-873d-4fe3-9454-81b9c5460b7f@192.168.203.25@tcp:311/0 lens 560/2880 e 0 to 0 dl 1767324696 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9883.025174] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:2060 to 0x280000401:2081) [ 9883.025230] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:2028 to 0x2c0000401:2049) [ 9884.448336] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9885.080823] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9885.846938] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9890.419127] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 22:31:30 (1767324690) [ 9894.179952] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9895.018159] Lustre: Failing over lustre-MDT0001 [ 9895.023844] LustreError: 257059:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x240002ef9:0xa9:0x0].0x0 (ffff9bb5bd211200) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9895.034945] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.25@tcp (stopping) [ 9895.038198] Lustre: Skipped 13 previous similar messages [ 9901.113614] Lustre: server umount lustre-MDT0001 complete [ 9917.102571] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9917.105561] LDISKFS-fs (dm-1): recovery complete [ 9917.116841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9918.669094] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9922.552427] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:2092 to 0x280000401:2113) [ 9922.552465] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:2060 to 0x2c0000401:2081) [ 9923.834767] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9924.494350] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9928.424255] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 22:32:08 (1767324728) [ 9931.819230] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9932.521438] Lustre: Failing over lustre-MDT0000 [ 9932.757620] Lustre: server umount lustre-MDT0000 complete [ 9932.767981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9932.773630] LustreError: Skipped 12 previous similar messages [ 9945.055180] Lustre: 257830:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767324731/real 1767324731] req@ffff9bb5ab09dc00 x1853164009396480/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1744/4320 e 0 to 1 dl 1767324747 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 9945.065418] Lustre: 257830:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 29 previous similar messages [ 9949.379230] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9949.380582] LDISKFS-fs (dm-0): recovery complete [ 9949.384454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9950.904914] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9954.818168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8289) [ 9954.818184] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8257) [ 9956.122905] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9956.706069] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9960.329562] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 22:32:40 (1767324760) [ 9960.739832] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 9960.741753] Lustre: Skipped 5 previous similar messages [ 9963.436949] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9966.318599] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9966.972420] Lustre: Failing over lustre-MDT0000 [ 9967.317704] Lustre: server umount lustre-MDT0000 complete [ 9984.338991] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9984.340595] LDISKFS-fs (dm-0): recovery complete [ 9984.345304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9984.397277] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9984.401802] LustreError: Skipped 9 previous similar messages [ 9986.061118] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9989.639480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8321) [ 9989.639485] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8289) [ 9991.130440] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9991.780250] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9993.840724] Lustre: Failing over lustre-MDT0001 [ 9993.932617] Lustre: server umount lustre-MDT0001 complete [10009.375480] LDISKFS-fs (dm-1): 6 truncates cleaned up [10009.377755] LDISKFS-fs (dm-1): recovery complete [10009.385421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10010.899108] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10014.716608] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:2092 to 0x280000401:2145) [10014.716711] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:2060 to 0x2c0000401:2113) [10016.021775] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10016.673584] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10020.761951] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 22:33:41 (1767324821) [10023.999257] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10026.924189] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10027.626154] Lustre: Failing over lustre-MDT0000 [10027.896778] Lustre: server umount lustre-MDT0000 complete [10029.335698] LustreError: 15426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324831 with bad export cookie 5068692619639845636 [10029.336781] Lustre: Failing over lustre-MDT0001 [10029.340335] LustreError: 15426:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10035.168514] LustreError: 265716:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9bb484838e00 x1853164009474816/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [10035.177959] LustreError: 265716:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [10035.294846] Lustre: server umount lustre-MDT0001 complete [10052.008233] LDISKFS-fs (dm-0): 2 truncates cleaned up [10052.010498] LDISKFS-fs (dm-0): recovery complete [10052.016482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10053.150916] LDISKFS-fs (dm-1): 6 truncates cleaned up [10053.153374] LDISKFS-fs (dm-1): recovery complete [10053.160230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10054.111177] LustreError: 267079:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10054.114452] LustreError: 267079:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9bb484838380 x1853164009474944/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767324856 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10054.124868] LustreError: 267079:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10054.623473] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9bb48e655880 x1853164009475584/t0(0) o250->MGC192.168.203.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10054.632446] LustreError: 6990:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [10056.572873] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10056.664145] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10059.957577] Lustre: 267264:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bb48e657100 x1853163992237056/t60129542146(0) o36->7755806c-873d-4fe3-9454-81b9c5460b7f@192.168.203.25@tcp:513/0 lens 496/2888 e 0 to 0 dl 1767324898 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [10059.962840] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:2060 to 0x2c0000401:2145) [10059.967112] Lustre: 267264:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [10059.971855] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:2092 to 0x280000401:2177) [10071.452694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8353) [10071.452758] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8321) [10072.969709] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10073.559799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10074.118595] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10077.961718] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 22:34:38 (1767324878) [10078.369937] LustreError: 268223:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9bb582cd9f80 x1853163992252032/t64424509442(0) o36->7755806c-873d-4fe3-9454-81b9c5460b7f@192.168.203.25@tcp:506/0 lens 496/456 e 0 to 0 dl 1767324891 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [10078.378067] LustreError: 268223:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 5 previous similar messages [10081.395284] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10082.073953] Lustre: Failing over lustre-MDT0000 [10082.295658] Lustre: server umount lustre-MDT0000 complete [10094.969900] Lustre: lustre-MDT0001: Client 7755806c-873d-4fe3-9454-81b9c5460b7f (at 192.168.203.25@tcp) reconnecting [10094.972302] Lustre: Skipped 1 previous similar message [10098.683904] LDISKFS-fs (dm-0): 2 truncates cleaned up [10098.684976] LDISKFS-fs (dm-0): recovery complete [10098.687546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10100.138815] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10104.317757] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8353) [10104.317776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8385) [10105.567536] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10106.141894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10109.877368] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 22:35:10 (1767324910) [10113.417644] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10114.211509] Lustre: Failing over lustre-MDT0001 [10114.327554] Lustre: server umount lustre-MDT0001 complete [10129.697613] LDISKFS-fs (dm-1): 6 truncates cleaned up [10129.698917] LDISKFS-fs (dm-1): recovery complete [10129.702841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10131.194380] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10135.027464] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:2060 to 0x2c0000401:2177) [10135.027472] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:2092 to 0x280000401:2209) [10136.252673] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10136.825271] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10140.281888] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 22:35:40 (1767324940) [10143.199509] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10145.681120] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10146.309070] Lustre: Failing over lustre-MDT0000 [10146.454280] Lustre: server umount lustre-MDT0000 complete [10162.229267] LDISKFS-fs (dm-0): 2 truncates cleaned up [10162.231516] LDISKFS-fs (dm-0): recovery complete [10162.234942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10163.615692] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10167.808957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8417) [10167.809206] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8385) [10169.031152] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10169.615617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10171.564636] Lustre: Failing over lustre-MDT0001 [10171.655424] Lustre: server umount lustre-MDT0001 complete [10186.727106] LDISKFS-fs (dm-1): 6 truncates cleaned up [10186.728660] LDISKFS-fs (dm-1): recovery complete [10186.732873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10188.095502] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10191.861326] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:2092 to 0x280000401:2241) [10191.861326] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:2060 to 0x2c0000401:2209) [10192.955537] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10193.477736] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10197.001484] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 22:36:37 (1767324997) [10199.786673] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10202.254622] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10202.893942] Lustre: Failing over lustre-MDT0000 [10203.004976] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.25@tcp (stopping) [10203.008756] Lustre: Skipped 10 previous similar messages [10203.140500] Lustre: server umount lustre-MDT0000 complete [10204.351602] LustreError: 15424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325006 with bad export cookie 5068692619639853154 [10204.352814] Lustre: Failing over lustre-MDT0001 [10204.354810] LustreError: 15424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10204.450276] Lustre: server umount lustre-MDT0001 complete [10220.629602] LDISKFS-fs (dm-0): 2 truncates cleaned up [10220.631488] LDISKFS-fs (dm-0): recovery complete [10220.637717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10221.721091] LDISKFS-fs (dm-1): 6 truncates cleaned up [10221.722513] LDISKFS-fs (dm-1): recovery complete [10221.726635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10230.240376] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x46579d151d471a02 [10231.750846] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10231.857277] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10239.371970] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000401:2092 to 0x280000401:2273) [10239.371971] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:2060 to 0x2c0000401:2241) [10244.504517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8067 to 0x280000400:8449) [10244.504521] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:7971 to 0x2c0000400:8417) [10245.929332] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10246.506599] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10247.032798] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10250.594687] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 22:37:31 (1767325051) [10251.210164] Lustre: 280441:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 7755806c-873d-4fe3-9454-81b9c5460b7f at adminstrative request [10257.043050] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 22:37:37 (1767325057) [10259.183851] Lustre: Failing over lustre-MDT0000 [10259.371318] Lustre: server umount lustre-MDT0000 complete [10274.318517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10276.038796] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10279.970033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:8468 to 0x2c0000400:8513) [10279.970378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8501 to 0x280000400:8545) [10281.292274] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10281.937829] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10285.389460] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 22:38:05 (1767325085) [10289.961140] Lustre: Failing over lustre-OST0000 [10290.019796] Lustre: server umount lustre-OST0000 complete [10303.913028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10305.888429] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10309.028452] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10309.575465] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10313.203126] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 22:38:33 (1767325113) [10314.472445] Lustre: Failing over lustre-MDT0000 [10314.714620] Lustre: server umount lustre-MDT0000 complete [10317.211738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10318.722976] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10319.486862] Lustre: lustre-MDT0000: Denying connection for new client 00f17004-5ae9-418b-96a9-b48001c606d0 (at 192.168.203.25@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [10322.422803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:8468 to 0x2c0000400:8545) [10322.422894] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8501 to 0x280000400:8577) [10327.249868] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 22:38:47 (1767325127) [10330.332282] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [10331.243451] Lustre: Failing over lustre-OST0000 [10331.281179] Lustre: server umount lustre-OST0000 complete [10346.162916] LDISKFS-fs (dm-2): 3 truncates cleaned up [10346.164070] LDISKFS-fs (dm-2): recovery complete [10346.167378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10348.118333] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10351.333709] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10351.877062] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10355.314411] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 22:39:15 (1767325155) [10358.131847] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [10360.046419] Lustre: Failing over lustre-OST0000 [10360.064081] LustreError: 287345:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [10360.066574] LustreError: 287345:0:(obd_class.h:479:obd_check_dev()) Skipped 159 previous similar messages [10360.088638] Lustre: server umount lustre-OST0000 complete [10360.705241] LustreError: 57653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10360.710750] LustreError: 57653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 200 previous similar messages [10361.824297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10361.828615] Lustre: Skipped 60 previous similar messages [10375.114803] LDISKFS-fs (dm-2): 3 truncates cleaned up [10375.116298] LDISKFS-fs (dm-2): recovery complete [10375.119769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10375.165906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10375.169021] Lustre: Skipped 20 previous similar messages [10375.171806] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10375.173925] Lustre: Skipped 20 previous similar messages [10376.058462] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10376.061141] Lustre: Skipped 20 previous similar messages [10376.766192] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.25@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000400:8579 extent [0-4194303]: client csum 66f0621a, server csum a9105c71 [10376.806901] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [10376.807042] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10376.809219] Lustre: Skipped 68 previous similar messages [10376.811882] Lustre: Skipped 20 previous similar messages [10377.127651] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10380.281180] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10380.885955] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10384.259515] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 22:39:44 (1767325184) [10386.669944] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [10389.111521] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10391.006935] Lustre: Failing over lustre-MDT0000 [10391.345244] Lustre: server umount lustre-MDT0000 complete [10392.555830] LustreError: 15424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325194 with bad export cookie 5068692619639879943 [10392.558337] Lustre: Failing over lustre-OST0000 [10392.561051] LustreError: 15424:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10392.602690] Lustre: server umount lustre-OST0000 complete [10407.641716] LDISKFS-fs (dm-0): 2 truncates cleaned up [10407.642977] LDISKFS-fs (dm-0): recovery complete [10407.645774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10418.425291] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10422.353077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:8468 to 0x2c0000400:8577) [10432.597482] LDISKFS-fs (dm-2): 3 truncates cleaned up [10432.598813] LDISKFS-fs (dm-2): recovery complete [10432.602601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10434.547121] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10440.317886] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 22:40:40 (1767325240) [10444.755104] Lustre: Failing over lustre-OST0000 [10444.819848] Lustre: server umount lustre-OST0000 complete [10446.117135] Lustre: Failing over lustre-MDT0000 [10446.366062] Lustre: server umount lustre-MDT0000 complete [10460.732902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10462.019155] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10466.295469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:8468 to 0x2c0000400:8609) [10470.285259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10472.169472] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10472.886773] Lustre: lustre-OST0000: Denying connection for new client f51e33c1-29ec-406d-9fae-c3fa80759c28 (at 192.168.203.25@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [10472.892596] Lustre: Skipped 1 previous similar message [10477.945713] Lustre: lustre-OST0000: Denying connection for new client f51e33c1-29ec-406d-9fae-c3fa80759c28 (at 192.168.203.25@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [10483.065786] Lustre: lustre-OST0000: Denying connection for new client f51e33c1-29ec-406d-9fae-c3fa80759c28 (at 192.168.203.25@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [10493.305803] Lustre: lustre-OST0000: Denying connection for new client f51e33c1-29ec-406d-9fae-c3fa80759c28 (at 192.168.203.25@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [10493.310092] Lustre: Skipped 1 previous similar message [10513.785761] Lustre: lustre-OST0000: Denying connection for new client f51e33c1-29ec-406d-9fae-c3fa80759c28 (at 192.168.203.25@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:27 [10513.790827] Lustre: Skipped 3 previous similar messages [10541.500145] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [10541.501806] Lustre: 294520:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 00f17004-5ae9-418b-96a9-b48001c606d0@ [10541.504378] Lustre: 294520:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [10541.506803] Lustre: lustre-OST0000: disconnecting 1 stale clients [10541.513939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8628 to 0x280000400:8649) [10545.261554] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [10549.330540] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [10551.579722] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 22:42:32 (1767325352) [10553.175097] Lustre: Failing over lustre-OST0001 [10553.234135] Lustre: server umount lustre-OST0001 complete [10553.311390] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10553.314940] LustreError: Skipped 16 previous similar messages [10566.823922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10568.850636] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10572.929466] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 22:42:53 (1767325373) [10574.170995] Lustre: Failing over lustre-OST0000 [10574.210746] Lustre: server umount lustre-OST0000 complete [10587.984079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10589.586412] LustreError: 297547:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [10589.590982] LustreError: 297547:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 76 previous similar messages [10590.171315] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10595.297837] Lustre: *** cfs_fail_loc=715, val=40*** [10605.535247] Lustre: 6990:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767325391/real 1767325391] req@ffff9bb589494700 x1853164009870464/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767325407 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [10605.545151] Lustre: 6990:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 17 previous similar messages [10605.548315] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [10605.552089] Lustre: Skipped 1 previous similar message [10611.679112] Lustre: *** cfs_fail_loc=715, val=40*** [10611.680276] Lustre: Skipped 2 previous similar messages [10612.703117] Lustre: *** cfs_fail_loc=715, val=40*** [10612.704513] Lustre: Skipped 1 previous similar message [10620.895672] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:09 [10620.899635] Lustre: Skipped 2 previous similar messages [10627.039160] Lustre: *** cfs_fail_loc=715, val=40*** [10629.631154] LustreError: 297547:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [10629.634080] LustreError: 297547:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 76 previous similar messages [10631.079393] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10631.658528] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10635.284643] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 22:43:55 (1767325435) [10636.653147] Lustre: Failing over lustre-MDT0000 [10636.985950] Lustre: server umount lustre-MDT0000 complete [10651.677376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10651.711779] LustreError: MGC192.168.203.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10651.715491] LustreError: Skipped 8 previous similar messages [10653.140822] Lustre: DEBUG MARKER: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10657.253321] LustreError: 299272:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [10658.271200] Lustre: *** cfs_fail_loc=715, val=80*** [10658.272427] Lustre: Skipped 2 previous similar messages [10668.421103] Lustre: lustre-MDT0000: Client f51e33c1-29ec-406d-9fae-c3fa80759c28 (at 192.168.203.25@tcp) reconnected, waiting for 2 clients in recovery for 1:29 [10668.425778] Lustre: Skipped 1 previous similar message [10673.631555] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10674.655245] Lustre: *** cfs_fail_loc=715, val=80*** [10674.657535] Lustre: Skipped 1 previous similar message [10684.793873] Lustre: lustre-MDT0000: Client f51e33c1-29ec-406d-9fae-c3fa80759c28 (at 192.168.203.25@tcp) reconnected, waiting for 2 clients in recovery for 1:12 [10688.992155] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10691.039151] Lustre: *** cfs_fail_loc=715, val=80*** [10691.040279] Lustre: Skipped 1 previous similar message [10700.153813] Lustre: lustre-MDT0000: Client f51e33c1-29ec-406d-9fae-c3fa80759c28 (at 192.168.203.25@tcp) reconnected, waiting for 2 clients in recovery for 0:57 [10705.375431] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10716.537690] Lustre: lustre-MDT0000: Client f51e33c1-29ec-406d-9fae-c3fa80759c28 (at 192.168.203.25@tcp) reconnected, waiting for 2 clients in recovery for 0:40 [10721.759487] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10727.903188] Lustre: *** cfs_fail_loc=715, val=80*** [10727.904457] Lustre: Skipped 4 previous similar messages [10732.921746] Lustre: lustre-MDT0000: Client f51e33c1-29ec-406d-9fae-c3fa80759c28 (at 192.168.203.25@tcp) reconnected, waiting for 2 clients in recovery for 0:24 [10737.119768] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10737.335145] LustreError: 299272:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [10737.339961] Lustre: 299272:0:(ldlm_lib.c:2931:target_recovery_thread()) too long recovery - read logs [10737.342238] LustreError: dumping log to /tmp/lustre-log.1767325539.299272 [10737.402277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:8663 to 0x280000400:8681) [10737.402277] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000400:8622 to 0x2c0000400:8641) [10738.681472] Lustre: DEBUG MARKER: oleg325-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10739.190275] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10742.806301] Lustre: DEBUG MARKER: == replay-single test complete, duration 10466 sec ======= 22:45:43 (1767325543) [10743.328750] Lustre: DEBUG MARKER: === replay-single: start cleanup 22:45:43 (1767325543) === [10745.666082] Lustre: DEBUG MARKER: === replay-single: finish cleanup 22:45:46 (1767325546) === [10746.355762] Lustre: Failing over lustre-MDT0000 [10746.481297] Lustre: server umount lustre-MDT0000 complete [10763.172208] loop: module loaded [10764.241340] Lustre: DEBUG MARKER: auster : @@@@@@ FAIL: Restart of mds1 failed! [10964.345748] LustreError: 279942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10964.350597] LustreError: 279942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 333 previous similar messages [11565.023538] LustreError: 205498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11565.029944] LustreError: 205498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 601 previous similar messages [12165.497979] LustreError: 279942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12165.504398] LustreError: 279942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 596 previous similar messages [12766.585848] LustreError: 279942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12766.590205] LustreError: 279942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 598 previous similar messages [13367.673724] LustreError: 281828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13367.679800] LustreError: 281828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 599 previous similar messages [13968.761696] LustreError: 279947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13968.766199] LustreError: 279947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 599 previous similar messages