[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 922847013 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003065] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007981] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010162] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013086] *** VALIDATE selinux *** [ 0.023151] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028717] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030140] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031149] *** VALIDATE tmpfs *** [ 0.032496] *** VALIDATE proc *** [ 0.034250] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.037050] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038177] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040045] Spectre V2 : User space: Vulnerable [ 0.041019] Speculative Store Bypass: Vulnerable [ 0.044431] debug: unmapping init [mem 0xffffffff9d659000-0xffffffff9d660fff] [ 0.047287] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048918] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049029] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051014] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054013] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056382] rcu: Hierarchical SRCU implementation. [ 0.058926] smp: Bringing up secondary CPUs ... [ 0.059858] x86: Booting SMP configuration: [ 0.060035] .... node #0, CPUs: #1 #2 #3 [ 0.071677] smp: Brought up 1 node, 4 CPUs [ 0.073018] smpboot: Max logical packages: 1 [ 0.074015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127019] node 0 deferred pages initialised in 51ms [ 0.130033] devtmpfs: initialized [ 0.131317] x86/mm: Memory block size: 128MB [ 0.132129] gcov: version magic: 0x41383552 [ 0.135352] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139179] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142465] pinctrl core: initialized pinctrl subsystem [ 0.145212] [ 0.145763] ************************************************************* [ 0.148014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150012] ** ** [ 0.153013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155012] ** ** [ 0.157015] ** This means that this kernel is built to expose internal ** [ 0.159012] ** IOMMU data structures, which may compromise security on ** [ 0.161017] ** your system. ** [ 0.162012] ** ** [ 0.165015] ** If you see this message and you are not debugging the ** [ 0.167011] ** kernel, report this immediately to your vendor! ** [ 0.170018] ** ** [ 0.172013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175013] ************************************************************* [ 0.177000] NET: Registered protocol family 16 [ 0.178547] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.181092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.183082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187241] cpuidle: using governor menu [ 0.189000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191666] PCI: Using configuration type 1 for base access [ 0.194172] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.205102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206053] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209278] cryptd: max_cpu_qlen set to 1000 [ 0.210287] ACPI: Added _OSI(Module Device) [ 0.211000] ACPI: Added _OSI(Processor Device) [ 0.212027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.220950] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229782] ACPI: Interpreter enabled [ 0.231066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233016] ACPI: Using IOAPIC for interrupt routing [ 0.235222] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.238607] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.255000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262034] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.267137] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.279079] acpiphp: Slot [2] registered [ 0.281196] acpiphp: Slot [5] registered [ 0.283175] acpiphp: Slot [6] registered [ 0.285205] acpiphp: Slot [7] registered [ 0.287175] acpiphp: Slot [8] registered [ 0.289134] acpiphp: Slot [9] registered [ 0.290107] acpiphp: Slot [10] registered [ 0.292306] acpiphp: Slot [3] registered [ 0.294156] acpiphp: Slot [4] registered [ 0.296142] acpiphp: Slot [11] registered [ 0.298142] acpiphp: Slot [12] registered [ 0.300272] acpiphp: Slot [13] registered [ 0.301130] acpiphp: Slot [14] registered [ 0.303123] acpiphp: Slot [15] registered [ 0.305127] acpiphp: Slot [16] registered [ 0.307141] acpiphp: Slot [17] registered [ 0.309107] acpiphp: Slot [18] registered [ 0.310261] acpiphp: Slot [19] registered [ 0.312102] acpiphp: Slot [20] registered [ 0.314096] acpiphp: Slot [21] registered [ 0.315282] acpiphp: Slot [22] registered [ 0.317112] acpiphp: Slot [23] registered [ 0.318289] acpiphp: Slot [24] registered [ 0.320135] acpiphp: Slot [25] registered [ 0.322119] acpiphp: Slot [26] registered [ 0.324119] acpiphp: Slot [27] registered [ 0.326119] acpiphp: Slot [28] registered [ 0.327096] acpiphp: Slot [29] registered [ 0.329120] acpiphp: Slot [30] registered [ 0.331556] acpiphp: Slot [31] registered [ 0.332105] PCI host bridge to bus 0000:00 [ 0.336034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.338023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.341030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.344030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.347036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.351035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.353188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357320] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361518] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.374018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.378049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.380191] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.383108] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.386033] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.389716] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.391177] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.394051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.398612] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.407021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.427029] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.433020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.442377] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.449020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.457025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.471021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.481000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.487019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.493025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.511036] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.523000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.529022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.536514] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.552041] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.555000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.567019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.574032] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.591019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.604222] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.611021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.622023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.645019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.662232] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.664016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.674018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.695017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.724177] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.726409] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.730514] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.733545] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.736316] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.743038] iommu: Default domain type: Passthrough [ 0.745421] SCSI subsystem initialized [ 0.746174] ACPI: bus type USB registered [ 0.749113] usbcore: registered new interface driver usbfs [ 0.751081] usbcore: registered new interface driver hub [ 0.752068] usbcore: registered new device driver usb [ 0.753182] pps_core: LinuxPPS API ver. 1 registered [ 0.756014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.759054] PTP clock support registered [ 0.762062] EDAC MC: Ver: 3.0.0 [ 0.770238] PCI: Using ACPI for IRQ routing [ 0.779249] NetLabel: Initializing [ 0.782016] NetLabel: domain hash size = 128 [ 0.784016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.905118] NetLabel: unlabeled traffic allowed by default [ 0.910147] vgaarb: loaded [ 0.911244] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.914015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.922232] clocksource: Switched to clocksource kvm-clock [ 1.439324] VFS: Disk quotas dquot_6.6.0 [ 1.442034] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.448059] *** VALIDATE ramfs *** [ 1.452337] *** VALIDATE hugetlbfs *** [ 1.456483] pnp: PnP ACPI init [ 1.464252] pnp: PnP ACPI: found 6 devices [ 1.513506] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.519634] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.525812] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.530442] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.536597] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.540329] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.546510] NET: Registered protocol family 2 [ 1.551945] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.565821] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.575547] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.587649] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.597552] TCP: Hash tables configured (established 65536 bind 65536) [ 1.606846] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.614598] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.621681] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.628306] NET: Registered protocol family 1 [ 1.636399] RPC: Registered named UNIX socket transport module. [ 1.638474] RPC: Registered udp transport module. [ 1.643935] RPC: Registered tcp transport module. [ 1.648430] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.652271] NET: Registered protocol family 44 [ 1.655422] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.660769] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.663271] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.667511] PCI: CLS 0 bytes, default 64 [ 1.671699] Unpacking initramfs... [ 4.819784] debug: unmapping init [mem 0xffff9abbbcc54000-0xffff9abbbffbffff] [ 4.829768] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.834645] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.839440] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.950504] Initialise system trusted keyrings [ 5.951777] Key type blacklist registered [ 5.953418] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.962354] zbud: loaded [ 5.964822] *** VALIDATE nfs *** [ 5.966330] *** VALIDATE nfs4 *** [ 5.967406] pstore: using deflate compression [ 5.970755] Platform Keyring initialized [ 6.319821] NET: Registered protocol family 38 [ 6.321769] Key type asymmetric registered [ 6.323136] Asymmetric key parser 'x509' registered [ 6.325383] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.329361] io scheduler mq-deadline registered [ 6.331321] io scheduler kyber registered [ 6.333249] io scheduler bfq registered [ 6.334959] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.338239] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.341421] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.344379] ACPI: Power Button [PWRF] [ 6.351640] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.359313] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.396826] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 6.403949] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.418643] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.448355] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.490486] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.499633] Non-volatile memory driver v1.3 [ 6.502811] Linux agpgart interface v0.103 [ 6.567257] virtio_blk virtio1: [vda] 68000 512-byte logical blocks (34.8 MB/33.2 MiB) [ 6.573153] vda: detected capacity change from 0 to 34816000 [ 6.598882] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.602740] vdb: detected capacity change from 0 to 1073741824 [ 6.622290] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.626330] vdc: detected capacity change from 0 to 2621440000 [ 6.645189] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.651789] vdd: detected capacity change from 0 to 2621440000 [ 6.673824] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.679344] vde: detected capacity change from 0 to 4294967296 [ 6.718475] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.721670] vdf: detected capacity change from 0 to 4294967296 [ 6.728898] libphy: Fixed MDIO Bus: probed [ 6.736387] usbcore: registered new interface driver usbserial_generic [ 6.738796] usbserial: USB Serial support registered for generic [ 6.741364] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.746571] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.748338] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.750425] mousedev: PS/2 mouse device common for all mice [ 6.752657] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.755483] rtc_cmos 00:05: RTC can wake from S4 [ 6.758389] rtc_cmos 00:05: registered as rtc0 [ 6.759560] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.764522] intel_pstate: CPU model not supported [ 6.767642] hid: raw HID events driver (C) Jiri Kosina [ 6.769840] usbcore: registered new interface driver usbhid [ 6.771810] usbhid: USB HID core driver [ 6.773182] drop_monitor: Initializing network drop monitor service [ 6.775784] Initializing XFRM netlink socket [ 6.777545] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.779576] NET: Registered protocol family 10 [ 6.786255] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.794545] Segment Routing with IPv6 [ 6.796813] NET: Registered protocol family 17 [ 6.798707] mpls_gso: MPLS GSO support [ 6.804412] RAS: Correctable Errors collector initialized. [ 6.806773] AVX version of gcm_enc/dec engaged. [ 6.808553] AES CTR mode by8 optimization enabled [ 6.922296] sched_clock: Marking stable (6922276767, 0)->(8024884973, -1102608206) [ 6.925692] registered taskstats version 1 [ 6.927782] Loading compiled-in X.509 certificates [ 6.930264] zswap: loaded using pool lzo/zbud [ 6.956358] Key type big_key registered [ 6.971778] Key type encrypted registered [ 6.973720] ima: No TPM chip found, activating TPM-bypass! [ 6.976345] ima: Allocated hash algorithm: sha1 [ 6.977939] ima: No architecture policies found [ 6.980059] evm: Initialising EVM extended attributes: [ 6.983629] evm: security.selinux [ 6.985377] evm: security.ima [ 6.986742] evm: security.capability [ 6.988572] evm: HMAC attrs: 0x1 [ 6.991592] rtc_cmos 00:05: setting system clock to 2026-02-24 21:28:49 UTC (1771968529) [ 7.002705] debug: unmapping init [mem 0xffffffff9e603000-0xffffffff9e7fffff] [ 7.007427] debug: unmapping init [mem 0xffffffff9d382000-0xffffffff9d658fff] [ 7.017106] Write protecting the kernel read-only data: 28672k [ 7.021873] debug: unmapping init [mem 0xffffffff9ba03000-0xffffffff9bbfffff] [ 7.025188] debug: unmapping init [mem 0xffffffff9c314000-0xffffffff9c3fffff] [ 7.078701] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.094723] systemd[1]: Detected virtualization kvm. [ 7.096989] systemd[1]: Detected architecture x86-64. [ 7.100671] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.127398] systemd[1]: No hostname configured. [ 7.130506] systemd[1]: Set hostname to . [ 7.132347] random: systemd: uninitialized urandom read (16 bytes read) [ 7.134500] systemd[1]: Initializing machine ID from random generator. [ 7.290422] random: ln: uninitialized urandom read (6 bytes read) [ 7.498938] random: systemd: uninitialized urandom read (16 bytes read) [ 7.503310] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 7.509852] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.516706] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.606637] device-mapper: uevent: version 1.0.3 [ 9.612936] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 10.909367] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.886670] virtio_net virtio0 ens2: renamed from eth0 [ 12.117582] scsi host0: ata_piix [ 12.258155] scsi host1: ata_piix [ 12.261558] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.270330] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.500708] random: crng init done [ 17.504208] random: 7 urandom warning(s) missed due to ratelimiting [ 19.159727] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK [ 20.995397] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.540471] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.754396] SELinux: Disabled at runtime. [ 24.866576] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.890252] systemd[1]: Detected virtualization kvm. [ 24.898154] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.473928] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.480269] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.499510] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.513933] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.524521] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.549803] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.591481] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ 27.094350] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 28.055837] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.076606] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.248868] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.658630] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.716866] EDAC sbridge: Ver: 1.1.2 [ 32.963290] Key type dns_resolver registered [ 33.766543] NFS: Registering the id_resolver key type [ 33.774159] Key type id_resolver registered [ 33.775673] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg337-server login: [ 63.010753] libcfs: loading out-of-tree module taints kernel. [ 63.043256] alg: No test for adler32 (adler32-zlib) [ 63.798895] Key type ._llcrypt registered [ 63.800523] Key type .llcrypt registered [ 63.884593] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_hostid [ 71.362796] hrtimer: interrupt took 5786526 ns [ 72.472820] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 73.076040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.376817] Lustre: Lustre: Build Version: 2.15.8_1_g8959aaa [ 73.757355] LNet: Added LNI 192.168.203.137@tcp [8/256/0/180] [ 73.764140] LNet: Accept secure, port 988 [ 75.407113] Key type lgssc registered [ 76.106473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 84.764611] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.280915] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.159551] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.961199] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.999833] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.227927] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing load_modules_local [ 113.157510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.201499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 114.341668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 114.361692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 114.430270] Lustre: lustre-MDT0000: new disk, initializing [ 114.509551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 114.520940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 116.668809] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 123.666521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 123.711308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 123.765925] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 123.777676] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 123.783725] Lustre: Skipped 1 previous similar message [ 123.833092] Lustre: lustre-MDT0001: new disk, initializing [ 123.859649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 123.876851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 123.881199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 125.895761] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 132.357891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 132.390049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 132.393798] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 132.490145] Lustre: lustre-OST0000: new disk, initializing [ 132.492371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 132.528231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.234848] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 137.718660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 137.722067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 140.485399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 140.537849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 140.543498] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 140.592142] Lustre: lustre-OST0001: new disk, initializing [ 140.594608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 140.626965] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 143.083682] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 144.315323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 144.324689] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 149.666968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 152.910799] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 159.245542] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing check_logdir /tmp/testlogs/ [ 160.868763] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing yml_node [ 162.774181] Lustre: DEBUG MARKER: Client: 2.15.8.1 [ 163.888820] Lustre: DEBUG MARKER: MDS: 2.15.8.1 [ 165.037203] Lustre: DEBUG MARKER: OSS: 2.15.8.1 [ 165.934696] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Feb 24 16:31:27 EST 2026 [ 170.108912] Lustre: DEBUG MARKER: excepting tests: 59 [ 173.080202] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing check_config_client /mnt/lustre [ 181.466066] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 182.916739] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 184.558595] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 186.263216] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 16:31:47 (1771968707) [ 189.736922] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 190.714491] Lustre: Failing over lustre-MDT0000 [ 190.808809] Lustre: server umount lustre-MDT0000 complete [ 193.217704] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 195.552076] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 195.552469] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 195.557967] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 195.568214] Lustre: Skipped 1 previous similar message [ 198.323242] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 198.333500] LustreError: Skipped 3 previous similar messages [ 200.389549] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 202.595777] Lustre: 3341:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771968718/real 1771968718] req@00000000b3f6d0e0 x1858043742556800/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771968725 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 202.618183] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 205.489174] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 205.497315] LustreError: Skipped 8 previous similar messages [ 207.766722] LDISKFS-fs (dm-0): recovery complete [ 207.768911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.883312] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe3742837 to 0x7f65554fe3742ee3 [ 208.890437] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 209.053109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.547730] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 211.381417] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 214.501208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 214.515549] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 217.228194] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 218.073191] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 222.263748] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 16:32:23 (1771968743) [ 223.177692] Lustre: Failing over lustre-OST0000 [ 223.194436] Lustre: server umount lustre-OST0000 complete [ 224.736202] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 224.736328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 224.740874] LustreError: Skipped 1 previous similar message [ 224.746050] Lustre: Skipped 2 previous similar messages [ 224.746836] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 224.755309] LustreError: Skipped 10 previous similar messages [ 237.141730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.221105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.259083] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 238.974938] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 238.975380] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 238.985476] Lustre: Skipped 3 previous similar messages [ 238.994057] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 239.075854] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 243.917373] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 244.723102] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 248.937824] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 16:32:50 (1771968770) [ 252.196109] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 253.099080] Lustre: Failing over lustre-MDT0000 [ 253.178851] Lustre: server umount lustre-MDT0000 complete [ 255.455462] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 255.459158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 255.464597] Lustre: Skipped 1 previous similar message [ 255.466725] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 255.472178] LustreError: Skipped 6 previous similar messages [ 258.016226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 258.024040] Lustre: Skipped 1 previous similar message [ 265.695147] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771968780/real 1771968780] req@0000000042efe66f x1858043742579776/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771968787 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 265.705093] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 268.485294] LDISKFS-fs (dm-0): recovery complete [ 268.487340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 271.840628] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe3742ee3 to 0x7f65554fe374390f [ 271.846336] Lustre: MGC192.168.203.137@tcp: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 271.849379] Lustre: Skipped 1 previous similar message [ 271.925555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 273.564127] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 274.487571] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 274.491158] Lustre: lustre-MDT0000: Denying connection for new client cf645d13-9498-4069-9efd-de5691976cb7 (at 192.168.203.37@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 276.961266] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 279.729231] Lustre: lustre-MDT0000: Denying connection for new client cf645d13-9498-4069-9efd-de5691976cb7 (at 192.168.203.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 284.850782] Lustre: lustre-MDT0000: Denying connection for new client cf645d13-9498-4069-9efd-de5691976cb7 (at 192.168.203.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 289.969416] Lustre: lustre-MDT0000: Denying connection for new client cf645d13-9498-4069-9efd-de5691976cb7 (at 192.168.203.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:50 [ 295.088735] Lustre: lustre-MDT0000: Denying connection for new client cf645d13-9498-4069-9efd-de5691976cb7 (at 192.168.203.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:44 [ 305.330329] Lustre: lustre-MDT0000: Denying connection for new client cf645d13-9498-4069-9efd-de5691976cb7 (at 192.168.203.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:34 [ 305.338096] Lustre: Skipped 1 previous similar message [ 325.809412] Lustre: lustre-MDT0000: Denying connection for new client cf645d13-9498-4069-9efd-de5691976cb7 (at 192.168.203.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:14 [ 325.817140] Lustre: Skipped 3 previous similar messages [ 340.000231] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 340.003519] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 340.010897] Lustre: lustre-MDT0000: Recovery over after 1:06, of 2 clients 1 recovered and 1 was evicted. [ 340.010947] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 340.021645] Lustre: Skipped 2 previous similar messages [ 340.033963] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:97 [ 340.034465] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:65 [ 345.570234] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 16:34:26 (1771968866) [ 348.982946] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 349.949161] Lustre: Failing over lustre-MDT0000 [ 350.030600] Lustre: server umount lustre-MDT0000 complete [ 350.178327] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 350.183150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 350.187429] Lustre: Skipped 1 previous similar message [ 350.189571] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.194758] LustreError: Skipped 20 previous similar messages [ 360.863126] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771968876/real 1771968876] req@00000000fdd6018d x1858043742604672/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771968883 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 360.872408] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 365.014365] LDISKFS-fs (dm-0): recovery complete [ 365.016502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 367.072975] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe374390f to 0x7f65554fe3743d14 [ 367.078575] LustreError: 19244:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.137@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000958e0b12) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 367.084851] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 367.163959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.729857] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 369.554390] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 369.557605] Lustre: lustre-MDT0000: Denying connection for new client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 369.562599] Lustre: Skipped 2 previous similar messages [ 435.000201] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 435.002955] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 435.008944] Lustre: lustre-MDT0000: Recovery over after 1:06, of 2 clients 1 recovered and 1 was evicted. [ 435.009383] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 435.018620] Lustre: Skipped 3 previous similar messages [ 435.028626] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:97 [ 435.030608] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:129 [ 440.253283] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 16:36:01 (1771968961) [ 443.014534] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 443.745074] Lustre: Failing over lustre-MDT0000 [ 443.872785] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 443.872958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 443.876618] Lustre: Skipped 4 previous similar messages [ 443.908911] Lustre: server umount lustre-MDT0000 complete [ 446.648823] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.656456] LustreError: Skipped 20 previous similar messages [ 456.159133] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771968971/real 1771968971] req@00000000ad26d76c x1858043742627968/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771968978 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 456.167371] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 458.848791] LDISKFS-fs (dm-0): recovery complete [ 458.850554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 462.303790] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe3743d14 to 0x7f65554fe37441b3 [ 462.385406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.792681] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 463.976696] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 467.440700] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 467.461070] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:129 [ 467.461338] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:161 [ 469.737556] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 470.480397] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 474.169528] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 16:36:35 (1771968995) [ 477.032174] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 477.798324] Lustre: Failing over lustre-MDT0000 [ 477.919556] Lustre: server umount lustre-MDT0000 complete [ 482.784142] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 482.784400] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.792011] Lustre: Skipped 5 previous similar messages [ 490.463166] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969005/real 1771969005] req@00000000d3ddc5eb x1858043742640128/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771969012 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 490.472188] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 492.918535] LDISKFS-fs (dm-0): recovery complete [ 492.921092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.583991] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe37441b3 to 0x7f65554fe3744698 [ 496.818403] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 497.194505] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 500.705196] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 500.707040] Lustre: Skipped 6 previous similar messages [ 500.728510] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 500.743248] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 500.743684] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:161 [ 502.694094] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 503.362984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 506.864181] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 16:37:08 (1771969028) [ 509.456374] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 510.114718] Lustre: Failing over lustre-MDT0000 [ 510.281615] Lustre: server umount lustre-MDT0000 complete [ 510.944199] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 523.231145] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969038/real 1771969038] req@000000000a308d62 x1858043742651392/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771969045 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 523.240782] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 524.883455] LDISKFS-fs (dm-0): recovery complete [ 524.885694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.376166] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe3744698 to 0x7f65554fe3744be6 [ 530.919474] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 531.389223] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 534.537979] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 534.556709] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:193 [ 534.556714] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 537.042031] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 537.761726] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 541.525440] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 16:37:42 (1771969062) [ 544.310663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 545.007764] Lustre: Failing over lustre-MDT0000 [ 545.150447] Lustre: server umount lustre-MDT0000 complete [ 549.855621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.856044] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 549.858794] Lustre: Skipped 7 previous similar messages [ 557.007090] Lustre: 3338:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969072/real 1771969072] req@00000000f8ff0186 x1858043742663680/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771969079 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 557.021060] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.192556] LDISKFS-fs (dm-0): recovery complete [ 560.194087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 563.168194] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe3744be6 to 0x7f65554fe37450ee [ 564.790965] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 568.319871] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 568.334626] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 568.334626] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:225 [ 570.370967] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 570.983276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 574.584343] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 16:38:15 (1771969095) [ 577.379557] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 578.528525] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.532793] LustreError: Skipped 102 previous similar messages [ 590.815131] Lustre: 3338:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969106/real 1771969106] req@00000000a78e8d34 x1858043742676096/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771969113 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 590.825290] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.456491] LDISKFS-fs (dm-0): recovery complete [ 593.458248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.959815] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe37450ee to 0x7f65554fe3745627 [ 597.049209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.052625] Lustre: Skipped 3 previous similar messages [ 598.193843] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 598.197439] Lustre: Skipped 1 previous similar message [ 598.697850] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 602.116579] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:257 [ 602.116579] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:289 [ 604.285551] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 604.895280] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 608.580521] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 16:38:49 (1771969129) [ 608.942249] Lustre: *** cfs_fail_loc=13b, val=315*** [ 608.943964] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 608.945919] LustreError: 16068:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@0000000083ba55a0 x1858043734191296/t38654705666(0) o35->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:747/0 lens 392/456 e 0 to 0 dl 1771969137 ref 1 fl Interpret:/0/0 rc 0/0 job:'openfile.0' [ 612.683592] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 613.354847] Lustre: Failing over lustre-MDT0000 [ 613.356060] Lustre: Skipped 1 previous similar message [ 613.487352] Lustre: server umount lustre-MDT0000 complete [ 613.489209] Lustre: Skipped 1 previous similar message [ 617.440134] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 617.440591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.448609] Lustre: Skipped 7 previous similar messages [ 628.362529] LDISKFS-fs (dm-0): recovery complete [ 628.364200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.240657] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 630.243774] Lustre: Skipped 18 previous similar messages [ 630.323736] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 631.900517] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 635.374962] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 635.378958] Lustre: Skipped 1 previous similar message [ 635.382150] Lustre: 16068:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000f2be3b3d x1858043734191296/t38654705666(0) o35->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:18/0 lens 392/456 e 0 to 0 dl 1771969163 ref 1 fl Interpret:/2/0 rc 0/0 job:'openfile.0' [ 635.395786] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:321 [ 635.395827] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:289 [ 637.784077] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 638.508395] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 642.222787] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 16:39:23 (1771969163) [ 645.025135] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 658.400105] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969173/real 1771969173] req@00000000c3b2e28e x1858043742702656/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771969180 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 658.409936] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 658.412760] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 658.417314] LustreError: Skipped 1 previous similar message [ 660.582410] LDISKFS-fs (dm-0): recovery complete [ 660.584520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.521168] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe3745afe to 0x7f65554fe3745fdc [ 663.524587] Lustre: Skipped 1 previous similar message [ 663.596426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 664.572826] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 664.575842] Lustre: Skipped 1 previous similar message [ 665.128303] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 668.670439] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:321 [ 668.670440] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:353 [ 670.586827] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 671.147023] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 674.628408] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 16:39:56 (1771969196) [ 677.354428] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 677.671311] Lustre: *** cfs_fail_loc=114, val=0*** [ 693.696678] LDISKFS-fs (dm-0): recovery complete [ 693.698347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.877445] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 698.389600] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 701.943637] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:385 [ 701.943638] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:353 [ 703.714253] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 704.293558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 707.910363] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 16:40:29 (1771969229) [ 710.393517] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 710.711296] Lustre: *** cfs_fail_loc=128, val=0*** [ 712.160824] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 712.164330] LustreError: Skipped 2 previous similar messages [ 726.643381] LDISKFS-fs (dm-0): recovery complete [ 726.645155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.225933] LustreError: 37610:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.137@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000091d45cfd) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 736.345928] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 737.930971] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 741.375662] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:385 [ 741.376265] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:417 [ 743.399394] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 743.940464] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 747.421623] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 16:41:08 (1771969268) [ 749.986962] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 750.741530] Lustre: Failing over lustre-MDT0000 [ 750.743568] Lustre: Skipped 3 previous similar messages [ 750.915626] Lustre: server umount lustre-MDT0000 complete [ 750.917238] Lustre: Skipped 3 previous similar messages [ 751.584789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.590364] Lustre: Skipped 14 previous similar messages [ 765.861240] LDISKFS-fs (dm-0): recovery complete [ 765.863089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 776.241089] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 777.785461] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 781.333935] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 781.336644] Lustre: Skipped 3 previous similar messages [ 781.351881] Lustre: lustre-OST0000: deleting orphan objects from 0x0:423 to 0x0:449 [ 781.351880] Lustre: lustre-OST0001: deleting orphan objects from 0x0:391 to 0x0:417 [ 783.347370] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 783.901276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 787.604599] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 16:41:49 (1771969309) [ 790.564260] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 803.807095] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969319/real 1771969319] req@000000009216b919 x1858043742754112/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771969326 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 803.815395] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 803.818509] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 803.822270] LustreError: Skipped 3 previous similar messages [ 806.296531] LDISKFS-fs (dm-0): recovery complete [ 806.298443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 809.951945] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe3746fa3 to 0x7f65554fe3747a31 [ 809.955754] Lustre: Skipped 3 previous similar messages [ 809.958140] LustreError: 41682:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.137@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000008bff73cc) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 810.041473] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 810.672453] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 810.675586] Lustre: Skipped 3 previous similar messages [ 811.428614] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 815.112320] Lustre: lustre-OST0001: deleting orphan objects from 0x0:423 to 0x0:449 [ 815.112323] Lustre: lustre-OST0000: deleting orphan objects from 0x0:455 to 0x0:481 [ 816.904626] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 817.442150] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 820.778165] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 16:42:22 (1771969342) [ 823.224031] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 825.312428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 825.315485] Lustre: Skipped 6 previous similar messages [ 835.551594] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 835.556024] LustreError: Skipped 215 previous similar messages [ 839.697789] LDISKFS-fs (dm-0): recovery complete [ 839.699344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.794751] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 844.188745] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 848.516534] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:577 [ 848.516553] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:609 [ 850.388220] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 851.001923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 860.588273] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 16:43:02 (1771969382) [ 863.245706] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 864.223575] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 864.226894] LustreError: Skipped 3 previous similar messages [ 878.558823] LDISKFS-fs (dm-0): recovery complete [ 878.560723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.186490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.188693] Lustre: Skipped 7 previous similar messages [ 881.197431] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 882.469932] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 886.245646] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 886.247969] Lustre: Skipped 35 previous similar messages [ 886.267336] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:609 [ 886.267344] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:641 [ 888.036711] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 888.531762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 893.710640] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 16:43:35 (1771969415) [ 896.097293] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 911.537526] LDISKFS-fs (dm-0): recovery complete [ 911.539236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.878396] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 920.574108] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:673 [ 920.574806] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:641 [ 922.459230] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 923.028382] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 926.645221] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 16:44:08 (1771969448) [ 929.302313] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 944.669474] LDISKFS-fs (dm-0): recovery complete [ 944.670820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.427327] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 955.429111] Lustre: Skipped 1 previous similar message [ 956.895388] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 960.509437] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:705 [ 960.509923] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:673 [ 962.357539] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 962.847689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 966.122083] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 16:44:47 (1771969487) [ 968.510461] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 983.754724] LDISKFS-fs (dm-0): recovery complete [ 983.756430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.298622] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 999.940061] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:705 [ 999.940066] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:737 [ 1001.724135] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1002.299583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1005.571609] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 16:45:27 (1771969527) [ 1007.886225] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1008.471997] Lustre: Failing over lustre-MDT0000 [ 1008.474045] Lustre: Skipped 6 previous similar messages [ 1008.596229] Lustre: server umount lustre-MDT0000 complete [ 1008.597477] Lustre: Skipped 6 previous similar messages [ 1010.143688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.147571] Lustre: Skipped 27 previous similar messages [ 1022.634085] LDISKFS-fs (dm-0): recovery complete [ 1022.636318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.787523] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1028.602285] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:769 [ 1028.602326] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:737 [ 1030.259510] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1030.751840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1033.870324] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 16:45:55 (1771969555) [ 1036.221275] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1051.361887] LDISKFS-fs (dm-0): recovery complete [ 1051.363906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.129514] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1057.774285] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1057.776731] Lustre: Skipped 7 previous similar messages [ 1057.792400] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:801 [ 1057.792400] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:769 [ 1059.589583] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1060.105128] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1063.434419] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 16:46:24 (1771969584) [ 1065.860937] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1075.167186] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771969590/real 1771969590] req@00000000bca6397b x1858043742896192/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771969597 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 1075.174769] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 1075.177610] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1075.181251] LustreError: Skipped 7 previous similar messages [ 1080.994971] LDISKFS-fs (dm-0): recovery complete [ 1080.996633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.311935] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe3752246 to 0x7f65554fe3752874 [ 1081.314703] Lustre: Skipped 7 previous similar messages [ 1082.032950] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1082.036155] Lustre: Skipped 7 previous similar messages [ 1082.862669] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1086.472532] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:833 [ 1086.472534] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:801 [ 1088.354763] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1088.912697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1092.294292] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 16:46:53 (1771969613) [ 1094.848860] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1109.679035] LDISKFS-fs (dm-0): recovery complete [ 1109.681333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.050647] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1110.053049] Lustre: Skipped 4 previous similar messages [ 1111.376093] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1115.125389] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:833 [ 1115.125694] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:865 [ 1116.882935] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1117.347677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1120.561602] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 16:47:22 (1771969642) [ 1122.975561] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1125.344110] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1125.347706] LustreError: Skipped 7 previous similar messages [ 1137.872528] LDISKFS-fs (dm-0): recovery complete [ 1137.873848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.009454] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1143.801035] Lustre: lustre-OST0001: deleting orphan objects from 0x0:835 to 0x0:865 [ 1143.801062] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:897 [ 1145.721470] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1146.211525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1149.436531] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 16:47:50 (1771969670) [ 1151.761524] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1166.849035] LDISKFS-fs (dm-0): recovery complete [ 1166.850735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.821372] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1172.469133] Lustre: lustre-OST0001: deleting orphan objects from 0x0:867 to 0x0:897 [ 1172.469443] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:929 [ 1174.300832] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1174.881142] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1178.159227] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 16:48:19 (1771969699) [ 1180.621576] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1195.840449] LDISKFS-fs (dm-0): recovery complete [ 1195.842050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.523938] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1201.150553] Lustre: lustre-OST0001: deleting orphan objects from 0x0:899 to 0x0:929 [ 1201.150553] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:961 [ 1202.989110] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1203.560537] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1206.875454] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 16:48:48 (1771969728) [ 1209.407863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1224.492101] LDISKFS-fs (dm-0): recovery complete [ 1224.493416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.862052] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1240.572196] Lustre: lustre-OST0001: deleting orphan objects from 0x0:899 to 0x0:961 [ 1240.572206] Lustre: lustre-OST0000: deleting orphan objects from 0x0:963 to 0x0:993 [ 1242.436899] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1242.974130] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1246.275060] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 16:49:27 (1771969767) [ 1248.621276] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1263.638156] LDISKFS-fs (dm-0): recovery complete [ 1263.639662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.550800] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1269.240138] Lustre: lustre-OST0000: deleting orphan objects from 0x0:963 to 0x0:1025 [ 1269.240182] Lustre: lustre-OST0001: deleting orphan objects from 0x0:963 to 0x0:993 [ 1271.032453] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1271.565175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1274.946222] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 16:49:56 (1771969796) [ 1277.436794] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1292.367777] LDISKFS-fs (dm-0): recovery complete [ 1292.369382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1304.436146] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1308.155826] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1025 [ 1308.155843] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 1310.070462] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1310.615662] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1314.097033] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 16:50:35 (1771969835) [ 1316.481607] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1331.581310] LDISKFS-fs (dm-0): recovery complete [ 1331.582784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.453571] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1347.074376] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1059 to 0x0:1089 [ 1347.074391] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 1349.095045] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1349.609064] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1352.918039] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 16:51:14 (1771969874) [ 1355.318056] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1357.280039] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.283524] LustreError: Skipped 443 previous similar messages [ 1370.513205] LDISKFS-fs (dm-0): recovery complete [ 1370.514675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.905731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1380.907674] Lustre: Skipped 7 previous similar messages [ 1382.254105] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1385.976459] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1059 to 0x0:1089 [ 1385.976461] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1059 to 0x0:1121 [ 1387.870409] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1388.371643] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1391.682909] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 16:51:53 (1771969913) [ 1392.952235] Lustre: 77333:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 68a40ec6-d031-4d33-a693-44634e1fc30a at adminstrative request [ 1409.794938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.600821] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 1413.602991] Lustre: Skipped 78 previous similar messages [ 1413.670161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.672307] Lustre: Skipped 15 previous similar messages [ 1415.014059] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1418.757737] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 1418.757810] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1059 to 0x0:1121 [ 1420.621060] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1421.189266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1423.632408] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1426.809536] Lustre: DEBUG MARKER: before 2844, after 2844 [ 1428.818197] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 16:52:30 (1771969950) [ 1429.160168] Lustre: 79362:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 68a40ec6-d031-4d33-a693-44634e1fc30a at adminstrative request [ 1433.805300] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 16:52:35 (1771969955) [ 1436.309616] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1451.615282] LDISKFS-fs (dm-0): recovery complete [ 1451.616655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.513428] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1457.157071] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 1457.157073] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1124 to 0x0:1153 [ 1459.072230] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1459.635148] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1462.992573] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 16:53:04 (1771969984) [ 1465.549792] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1480.704760] LDISKFS-fs (dm-0): recovery complete [ 1480.706540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.426164] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1497.078403] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 1497.078975] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 1499.049816] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1499.633849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1502.957730] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 16:53:44 (1771970024) [ 1505.577432] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1520.365921] LDISKFS-fs (dm-0): recovery complete [ 1520.367236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1532.322403] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1535.991106] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 1535.991112] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 1537.795019] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1538.340664] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1541.836128] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 16:54:23 (1771970063) [ 1544.317373] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1544.927739] Lustre: Failing over lustre-MDT0000 [ 1544.928874] Lustre: Skipped 15 previous similar messages [ 1545.066469] Lustre: server umount lustre-MDT0000 complete [ 1545.068061] Lustre: Skipped 15 previous similar messages [ 1546.207710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1546.213043] Lustre: Skipped 63 previous similar messages [ 1559.439402] LDISKFS-fs (dm-0): recovery complete [ 1559.441133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.258435] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1574.887578] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1574.891587] Lustre: Skipped 14 previous similar messages [ 1574.907920] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 1574.907931] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 1576.779287] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1577.323522] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1580.702298] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 16:55:02 (1771970102) [ 1583.283777] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1591.775090] Lustre: 3338:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771970107/real 1771970107] req@00000000cfc316eb x1858043743089408/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771970114 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1591.782099] Lustre: 3338:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 1591.784485] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1591.787243] LustreError: Skipped 14 previous similar messages [ 1598.601445] LDISKFS-fs (dm-0): recovery complete [ 1598.604406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.184390] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe37574a2 to 0x7f65554fe37579e9 [ 1609.187493] Lustre: Skipped 14 previous similar messages [ 1609.392591] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1609.395664] Lustre: Skipped 14 previous similar messages [ 1610.622056] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1614.324917] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1313 [ 1614.324921] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 1616.137671] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1616.700119] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1619.959225] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 16:55:41 (1771970141) [ 1622.455044] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1637.537718] LDISKFS-fs (dm-0): recovery complete [ 1637.539241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.856961] LustreError: 90913:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.137@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000009e5a97e4) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1639.332463] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1643.015287] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 1643.015313] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 1644.825788] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1645.339099] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1648.744693] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 16:56:10 (1771970170) [ 1651.247996] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1653.216114] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1653.218867] LustreError: Skipped 14 previous similar messages [ 1666.299723] LDISKFS-fs (dm-0): recovery complete [ 1666.301725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.629859] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1682.432309] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 1682.432326] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 1684.237661] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1684.807794] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1688.114044] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 16:56:49 (1771970209) [ 1690.541795] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1705.523940] LDISKFS-fs (dm-0): recovery complete [ 1705.525304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1707.484954] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1711.096543] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 1711.096545] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 1712.931894] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1713.477580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1717.026131] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 16:57:18 (1771970238) [ 1719.540857] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1734.606212] LDISKFS-fs (dm-0): recovery complete [ 1734.607508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.338942] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1750.008295] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 1750.008310] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 1751.868177] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1752.376741] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1755.646379] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 16:57:57 (1771970277) [ 1758.190105] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1773.274279] LDISKFS-fs (dm-0): recovery complete [ 1773.275928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.216052] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1788.920992] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 1788.921050] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 1790.734927] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1791.248967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1794.586804] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 16:58:36 (1771970316) [ 1797.139500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1812.123048] LDISKFS-fs (dm-0): recovery complete [ 1812.124471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.326452] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1818.101688] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 1818.102167] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1475 to 0x0:1505 [ 1819.897192] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1820.412654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1823.655180] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 16:59:05 (1771970345) [ 1823.992410] Lustre: 102015:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 68a40ec6-d031-4d33-a693-44634e1fc30a at adminstrative request [ 1828.577640] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 16:59:10 (1771970350) [ 1830.703244] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1835.537888] LDISKFS-fs (dm-0): recovery complete [ 1835.539340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.648097] LustreError: 103296:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 1835.650595] LustreError: 103296:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1835.653188] Lustre: 103329:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1835.655971] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1835.671487] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1475 to 0x0:1505 [ 1835.671487] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1475 to 0x0:1537 [ 1837.006361] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1841.120700] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1848.238725] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 16:59:29 (1771970369) [ 1850.653449] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1851.361284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.364654] Lustre: Skipped 3 previous similar messages [ 1854.129585] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 1856.479506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.481737] Lustre: Skipped 3 previous similar messages [ 1860.934437] LDISKFS-fs (dm-0): recovery complete [ 1860.935712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.044379] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1861.054278] LustreError: 105209:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 1861.056494] LustreError: 105209:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1861.058609] Lustre: 105242:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1861.062501] Lustre: 105242:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 1861.066422] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1861.082913] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1475 to 0x0:1569 [ 1861.090172] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1475 to 0x0:1537 [ 1862.564585] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1866.209137] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1871.759803] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1873.831808] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 16:59:55 (1771970395) [ 1876.148679] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1880.796579] LDISKFS-fs (dm-0): recovery complete [ 1880.798086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.911677] LustreError: 107124:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 1880.914394] LustreError: 107124:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1880.917015] Lustre: 107157:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1880.920579] Lustre: 107157:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 1880.923831] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1880.939920] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1601 [ 1880.940285] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1544 to 0x0:1569 [ 1882.280561] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1886.176594] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1894.322826] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 17:00:15 (1771970415) [ 1894.672991] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1894.674388] LustreError: 10003:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@000000005bc3df5b x1858043734660544/t201863462916(0) o36->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:523/0 lens 504/456 e 0 to 0 dl 1771970423 ref 1 fl Interpret:/0/0 rc 0/0 job:'rm.0' [ 1900.325451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.428956] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1900.429076] LustreError: 108497:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 1900.432696] Lustre: Skipped 21 previous similar messages [ 1900.435043] LustreError: 108497:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1900.440719] Lustre: 108530:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1900.443888] Lustre: 108530:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 1900.446894] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1900.462583] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1633 [ 1900.462601] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1571 to 0x0:1601 [ 1901.842107] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1905.633817] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1914.735505] Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ============ 17:00:36 (1771970436) [ 1917.144938] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1932.298954] LDISKFS-fs (dm-0): recovery complete [ 1932.300236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.818955] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1939.452808] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1665 [ 1939.452840] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1603 to 0x0:1633 [ 1941.675803] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 17:01:03 (1771970463) [ 1944.107966] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1949.305502] LDISKFS-fs (dm-0): recovery complete [ 1949.307192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.420985] LustreError: 112268:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 1949.423635] LustreError: 112268:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1949.426127] Lustre: 112301:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1949.429459] Lustre: 112301:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 1949.431563] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1949.446683] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1697 [ 1949.447300] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1603 to 0x0:1665 [ 1950.836872] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1954.785202] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1962.452272] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 17:01:23 (1771970483) [ 1969.511029] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1971.890089] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.893802] LustreError: Skipped 452 previous similar messages [ 1984.845235] LDISKFS-fs (dm-0): recovery complete [ 1984.846797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.052963] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1992.716867] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2098 to 0x0:2113 [ 1992.716869] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2066 to 0x0:2081 [ 1994.541972] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1995.095437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2002.513476] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 17:02:03 (1771970523) [ 2007.666146] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2024.375670] LDISKFS-fs (dm-0): recovery complete [ 2024.377074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.464906] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 2026.466562] Lustre: Skipped 94 previous similar messages [ 2026.550251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.553165] Lustre: Skipped 18 previous similar messages [ 2027.947822] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2032.118494] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2482 to 0x0:2497 [ 2032.122331] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2514 to 0x0:2529 [ 2033.998339] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2034.560525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2042.082922] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 17:02:43 (1771970563) [ 2042.674055] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 2043.249660] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 17:02:44 (1771970564) [ 2043.959323] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2044.276994] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2044.281767] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2044.285897] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2482 to 0x0:2529 [ 2046.438828] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 17:02:47 (1771970567) [ 2052.213419] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2069.023427] LDISKFS-fs (dm-2): recovery complete [ 2069.024678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2070.490835] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2071.043088] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:2977 [ 2071.043665] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 2115.309925] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 17:03:56 (1771970636) [ 2117.893723] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2133.050876] LDISKFS-fs (dm-0): recovery complete [ 2133.052059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.915199] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2138.617663] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2138.617921] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:2945 [ 2140.858397] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2141.573969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2145.759287] LustreError: 122019:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2145.761385] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2146.783816] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3009 [ 2155.165171] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 17:04:36 (1771970676) [ 2156.836546] LustreError: 17418:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2162.143205] LustreError: 17418:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2162.145397] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnecting [ 2162.742717] LustreError: 17418:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2167.775130] LustreError: 17418:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2167.777649] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnecting [ 2168.343046] LustreError: 17418:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2173.407093] LustreError: 17418:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2173.409227] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnecting [ 2173.994460] LustreError: 17418:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2179.039081] LustreError: 17418:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2179.646353] LustreError: 17418:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2184.671099] LustreError: 17418:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2184.673277] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnecting [ 2184.675824] Lustre: Skipped 1 previous similar message [ 2190.894340] LustreError: 17418:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2190.896577] LustreError: 17418:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message [ 2195.935106] LustreError: 17418:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2195.937036] LustreError: 17418:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message [ 2201.567139] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnecting [ 2201.570147] Lustre: Skipped 2 previous similar messages [ 2207.802830] LustreError: 6238:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 2207.804737] LustreError: 6238:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages [ 2212.831172] LustreError: 6238:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 2212.834069] LustreError: 6238:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages [ 2215.535308] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 17:05:37 (1771970737) [ 2216.156153] LustreError: 6238:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2222.256857] Lustre: lustre-MDT0000: Export 00000000e38bec30 already connecting from 192.168.203.37@tcp [ 2222.846773] Lustre: lustre-MDT0000: Export 00000000e38bec30 already connecting from 192.168.203.37@tcp [ 2224.012396] Lustre: lustre-MDT0000: Export 00000000e38bec30 already connecting from 192.168.203.37@tcp [ 2224.014442] Lustre: Skipped 1 previous similar message [ 2226.323514] Lustre: lustre-MDT0000: Export 00000000e38bec30 already connecting from 192.168.203.37@tcp [ 2226.325908] Lustre: Skipped 3 previous similar messages [ 2228.015047] LustreError: 6238:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 2229.441948] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 17:05:50 (1771970750) [ 2234.063855] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2235.120897] Lustre: Failing over lustre-MDT0000 [ 2235.122588] Lustre: Skipped 17 previous similar messages [ 2235.328837] Lustre: server umount lustre-MDT0000 complete [ 2235.330900] Lustre: Skipped 17 previous similar messages [ 2235.871438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.874674] Lustre: Skipped 69 previous similar messages [ 2239.360506] LDISKFS-fs (dm-0): recovery complete [ 2239.361748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.394796] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2239.398346] LustreError: Skipped 15 previous similar messages [ 2239.399974] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe378d4e3 to 0x7f65554fe378e559 [ 2239.402413] Lustre: Skipped 15 previous similar messages [ 2239.450406] Lustre: *** cfs_fail_loc=712, val=0*** [ 2239.452459] LustreError: 44651:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery req@000000009b4d52de x1858043743551040/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' [ 2239.462348] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 2239.476049] LustreError: 126684:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device [ 2239.480097] LustreError: 126684:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2239.482192] Lustre: 126719:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2239.484567] Lustre: 126719:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages [ 2239.486675] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2239.503638] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3041 [ 2239.503666] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:2977 [ 2240.825855] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2244.578868] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2254.815730] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2254.817922] LustreError: Skipped 18 previous similar messages [ 2261.983119] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771970777/real 1771970777] req@000000005988151a x1858043743558400/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771970784 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 2261.990383] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 2263.800939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.305432] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2269.309559] Lustre: Skipped 11 previous similar messages [ 2269.549561] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2273.253376] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2273.257104] Lustre: Skipped 12 previous similar messages [ 2273.274994] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3073 [ 2273.274997] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:3009 [ 2275.051489] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2275.576814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2279.010636] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 17:06:40 (1771970800) [ 2279.045735] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnecting [ 2279.048218] Lustre: Skipped 4 previous similar messages [ 2282.008640] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 17:06:43 (1771970803) [ 2282.297867] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2282.299564] LustreError: 102218:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@000000000ea19f50 x1858043735696512/t0(0) o700->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:155/0 lens 264/248 e 0 to 0 dl 1771970810 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0' [ 2303.384414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.511421] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2312.178062] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3011 to 0x0:3041 [ 2312.178161] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3075 to 0x0:3105 [ 2313.949292] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2314.498279] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2318.060648] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 17:07:19 (1771970839) [ 2331.642239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2333.025629] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2333.365953] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137 [ 2333.367087] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97 [ 2336.723192] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2337.255421] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2402.443435] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 17:08:43 (1771970923) [ 2404.921069] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2419.969495] LDISKFS-fs (dm-0): recovery complete [ 2419.970897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.990924] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2425.904221] Lustre: *** cfs_fail_loc=216, val=0*** [ 2425.904256] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169 [ 2425.905702] LustreError: 133113:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2426.975460] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3062 to 0x0:3105 [ 2488.657233] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 17:10:10 (1771971010) [ 2489.331266] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2489.335446] Lustre: Skipped 2 previous similar messages [ 2489.338452] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201 [ 2489.714407] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3233 [ 2496.653597] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 17:10:18 (1771971018) [ 2510.754609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.010653] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2516.014148] Lustre: Skipped 17 previous similar messages [ 2517.309510] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2521.067640] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2521.069557] LustreError: 134791:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000d5477521 x1858043735747456/t0(0) o101->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:405/0 lens 328/344 e 0 to 0 dl 1771971060 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0' [ 2538.160815] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 2538.185279] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3265 [ 2538.185355] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3117 to 0x0:3137 [ 2540.214917] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2540.757697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2544.267321] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 17:11:05 (1771971065) [ 2545.636198] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2548.142696] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2548.703436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.904817] LDISKFS-fs (dm-0): recovery complete [ 2562.906023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2565.088802] LustreError: 136972:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.137@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000023737b5b) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2566.498379] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2570.228118] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3297 [ 2570.228135] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3117 to 0x0:3169 [ 2572.038405] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2572.593138] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2575.861911] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 17:11:37 (1771971097) [ 2576.253442] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2579.829308] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2580.447871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2580.450350] Lustre: Skipped 3 previous similar messages [ 2580.657684] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.664343] LustreError: Skipped 218 previous similar messages [ 2594.695925] LDISKFS-fs (dm-0): recovery complete [ 2594.697363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.276341] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2604.025906] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3329 [ 2604.025906] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3201 [ 2605.617449] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2606.152529] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2609.546991] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 17:12:11 (1771971131) [ 2609.890675] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2613.658851] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2628.626536] LDISKFS-fs (dm-0): recovery complete [ 2628.627945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.649476] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 2631.653071] Lustre: Skipped 53 previous similar messages [ 2631.718182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2631.721555] Lustre: Skipped 10 previous similar messages [ 2633.109732] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2636.789684] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233 [ 2636.789741] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3361 [ 2641.132916] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 17:12:42 (1771971162) [ 2642.497625] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2642.499050] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2642.501164] LustreError: 6239:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000ce1d4382 x1858043735773184/t261993005073(0) o35->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:516/0 lens 392/456 e 0 to 0 dl 1771971171 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 2656.493720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2661.673236] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2665.455842] Lustre: 6239:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000023627942 x1858043735773184/t261993005073(0) o35->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:538/0 lens 392/456 e 0 to 0 dl 1771971193 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 2665.463822] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3393 [ 2665.463859] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3265 [ 2667.300982] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2667.799777] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2671.116230] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 17:13:12 (1771971192) [ 2671.466638] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2671.468079] LustreError: 7800:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000868cb528 x1858043735780224/t266287972368(0) o36->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:558/0 lens 504/448 e 0 to 0 dl 1771971213 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 2675.151580] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2689.990273] LDISKFS-fs (dm-0): recovery complete [ 2689.992537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2695.465665] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2699.248721] Lustre: 7800:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000039b822ff x1858043735780224/t266287972368(0) o36->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:586/0 lens 504/448 e 0 to 0 dl 1771971241 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 2699.254586] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3425 [ 2699.254630] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3297 [ 2701.030266] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2701.550374] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2704.969287] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 17:13:46 (1771971226) [ 2705.335785] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2705.337781] LustreError: 7800:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@0000000022856f36 x1858043735787648/t270582939664(0) o36->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:592/0 lens 504/448 e 0 to 0 dl 1771971247 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 2706.604250] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2708.970672] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2724.030701] LDISKFS-fs (dm-0): recovery complete [ 2724.032047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2729.410450] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2733.036853] Lustre: 7800:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000006bb0d064 x1858043735787648/t270582939664(0) o36->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:620/0 lens 504/448 e 0 to 0 dl 1771971275 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 2733.042385] Lustre: 7800:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 2733.045459] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3457 [ 2733.045464] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3329 [ 2737.351996] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 17:14:18 (1771971258) [ 2737.688699] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2737.690160] Lustre: Skipped 1 previous similar message [ 2737.691257] LustreError: 6237:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000a950a0f7 x1858043735794624/t274877906960(0) o36->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:625/0 lens 504/448 e 0 to 0 dl 1771971280 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 2737.696234] LustreError: 6237:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message [ 2738.969263] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2738.970763] Lustre: Skipped 1 previous similar message [ 2741.549957] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2756.328682] LDISKFS-fs (dm-0): recovery complete [ 2756.329835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.176222] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2771.949132] Lustre: 6238:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000d5b7e3e1 x1858043735794624/t274877906960(0) o36->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:659/0 lens 504/448 e 0 to 0 dl 1771971314 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 2771.953628] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3361 [ 2771.953628] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3489 [ 2775.892639] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 17:14:57 (1771971297) [ 2776.208374] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2777.493881] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2777.496136] LustreError: 6239:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000b8f316c9 x1858043735801856/t279172874256(0) o35->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:651/0 lens 392/456 e 0 to 0 dl 1771971306 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 2780.733731] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2795.403933] LDISKFS-fs (dm-0): recovery complete [ 2795.405218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.864740] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2800.620996] Lustre: 6239:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000a0039472 x1858043735801856/t279172874256(0) o35->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:674/0 lens 392/456 e 0 to 0 dl 1771971329 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 2800.626342] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521 [ 2800.626801] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3393 [ 2804.962258] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 17:15:26 (1771971326) [ 2805.292413] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 2805.295662] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2805.297828] Lustre: Skipped 1 previous similar message [ 2826.416470] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnecting [ 2826.421100] Lustre: Skipped 1 previous similar message [ 2826.424215] Lustre: 17418:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000849acdcb x1858043735807232/t283467841550(0) o101->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:713/0 lens 664/3424 e 0 to 0 dl 1771971368 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0' [ 2828.697630] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 17:15:50 (1771971350) [ 2831.288644] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2836.447526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.450742] Lustre: Skipped 57 previous similar messages [ 2842.591204] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2842.594317] LustreError: Skipped 12 previous similar messages [ 2846.334300] LDISKFS-fs (dm-0): recovery complete [ 2846.336131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.735985] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe37943d2 to 0x7f65554fe3794a38 [ 2848.740337] Lustre: Skipped 12 previous similar messages [ 2850.154044] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2853.884177] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3395 to 0x0:3425 [ 2853.884262] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3553 [ 2855.671102] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2856.198615] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2869.736395] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 17:16:31 (1771971391) [ 2872.397950] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2872.993761] Lustre: Failing over lustre-MDT0000 [ 2872.994833] Lustre: Skipped 14 previous similar messages [ 2873.178437] Lustre: server umount lustre-MDT0000 complete [ 2873.180626] Lustre: Skipped 14 previous similar messages [ 2874.335889] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2874.338167] LustreError: Skipped 13 previous similar messages [ 2881.503165] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771971396/real 1771971396] req@0000000099baee4f x1858043743776448/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771971403 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 2881.509200] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [ 2887.334137] LDISKFS-fs (dm-0): recovery complete [ 2887.335396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.856549] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2887.858765] Lustre: Skipped 13 previous similar messages [ 2888.974820] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2892.782166] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2892.784429] Lustre: Skipped 13 previous similar messages [ 2892.798820] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3585 [ 2892.798834] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3427 to 0x0:3457 [ 2894.556126] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2895.052950] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2897.394986] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2902.382314] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 17:17:03 (1771971423) [ 2911.049446] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2926.169732] LDISKFS-fs (dm-0): recovery complete [ 2926.171234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.208452] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2941.990988] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4708 to 0x0:4737 [ 2941.991070] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4865 [ 2943.733624] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2944.247176] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2960.496775] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 17:18:01 (1771971481) [ 2963.625859] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2978.683215] LDISKFS-fs (dm-0): recovery complete [ 2978.684626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.530988] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2985.986144] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4867 to 0x0:4897 [ 2985.986171] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4708 to 0x0:4769 [ 2987.935476] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2988.507971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2992.082657] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 2992.636500] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2994.690725] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 17:18:36 (1771971516) [ 3000.551065] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3022.845974] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3022.847813] LustreError: 7800:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@000000001efc18b5 x1858043737284160/t300647710728(0) o36->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:155/0 lens 66040/440 e 0 to 0 dl 1771971565 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0' [ 3022.853839] LustreError: 7800:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message [ 3044.018134] Lustre: 6238:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000fd0fe857 x1858043737284160/t300647710728(0) o36->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:176/0 lens 66040/440 e 0 to 0 dl 1771971586 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0' [ 3046.936290] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3047.510990] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 17:19:28 (1771971568) [ 3051.471635] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3066.540829] LDISKFS-fs (dm-0): recovery complete [ 3066.542210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.364941] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3076.211636] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4998 to 0x0:5025 [ 3076.211659] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4871 to 0x0:4897 [ 3077.963938] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3078.512459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3082.045056] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 17:20:03 (1771971603) [ 3087.315324] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3104.708379] LDISKFS-fs (dm-2): recovery complete [ 3104.709892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3106.139527] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3106.418980] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129 [ 3106.419934] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5441 [ 3130.432662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3130.488039] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3130.489833] Lustre: Skipped 14 previous similar messages [ 3131.839477] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3132.216567] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161 [ 3132.221553] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5473 [ 3135.530891] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3136.049286] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3170.072790] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 17:21:31 (1771971691) [ 3182.256681] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.262094] LustreError: Skipped 352 previous similar messages [ 3184.127083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.996481] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3189.766443] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5298 to 0x0:5313 [ 3189.766529] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5505 [ 3210.837180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.952314] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3218.426756] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5537 [ 3218.426777] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5298 to 0x0:5345 [ 3220.188460] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3220.702729] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3224.154057] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 17:22:25 (1771971745) [ 3248.584601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3248.641359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.644318] Lustre: Skipped 14 previous similar messages [ 3249.992428] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3250.352705] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.137@tcp (at 0@lo) [ 3250.354671] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:193 [ 3250.355167] Lustre: Skipped 68 previous similar messages [ 3250.360386] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5569 [ 3253.750241] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3254.226725] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3258.313404] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 17:22:59 (1771971779) [ 3259.359514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3263.685328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.725734] Lustre: *** cfs_fail_loc=605, val=0*** [ 3263.727118] LustreError: 171756:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=00000000645f21de failed: rc = -95 [ 3263.730332] LustreError: 171756:0:(obd_config.c:774:class_setup()) setup MGS failed (-95) [ 3263.733989] LustreError: 171756:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95 [ 3263.738210] LustreError: 171756:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered [ 3263.742947] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3263.746364] LustreError: 171756:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 3263.753158] LustreError: 171756:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3266.423146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.900356] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3270.580145] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 17:23:12 (1771971792) [ 3271.687981] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5347 to 0x0:5377 [ 3271.688165] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5601 [ 3274.325017] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3289.778928] LDISKFS-fs (dm-0): recovery complete [ 3289.780392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3290.802176] Lustre: *** cfs_fail_loc=707, val=0*** [ 3291.443517] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3311.287994] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 3311.387404] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5390 to 0x0:5409 [ 3311.387407] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5615 to 0x0:5633 [ 3313.342573] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3313.848232] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3317.496392] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 17:23:59 (1771971839) [ 3340.763340] LustreError: 6238:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 3346.799160] LustreError: 6238:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3358.677918] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 17:24:40 (1771971880) [ 3382.810568] LustreError: 39084:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms [ 3388.847070] LustreError: 39084:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake [ 3391.757469] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 17:25:13 (1771971913) [ 3414.684774] LustreError: 6236:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3419.783083] LustreError: 6236:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3420.501451] LustreError: 6236:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 3430.591071] LustreError: 6236:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3442.298216] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 17:26:03 (1771971963) [ 3490.953210] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 17:26:52 (1771972012) [ 3513.891075] LustreError: 17418:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 3514.311111] LustreError: 17418:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3522.015549] LustreError: 8214:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 3522.021378] LustreError: 8214:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 28 previous similar messages [ 3522.431065] LustreError: 8213:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3522.434018] LustreError: 8213:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 27 previous similar messages [ 3538.107165] LustreError: 6238:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 3538.111696] LustreError: 6238:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 63 previous similar messages [ 3538.535077] LustreError: 6238:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 3538.537119] LustreError: 6238:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 63 previous similar messages [ 3541.356860] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 17:27:42 (1771972062) [ 3565.885987] Lustre: DEBUG MARKER: phase 2 [ 3568.820645] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 17:28:10 (1771972090) [ 3639.591022] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 17:29:21 (1771972161) [ 3640.109642] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 3640.662538] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 17:29:22 (1771972162) [ 3642.370823] Lustre: DEBUG MARKER: Started rundbench load pid=138183 ... [ 3645.849917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3647.354814] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 3648.005766] Lustre: Failing over lustre-MDT0000 [ 3648.008028] Lustre: Skipped 10 previous similar messages [ 3648.028435] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 3648.261304] Lustre: server umount lustre-MDT0000 complete [ 3648.262565] Lustre: Skipped 11 previous similar messages [ 3648.479792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3648.483142] Lustre: Skipped 42 previous similar messages [ 3661.215091] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771972176/real 1771972176] req@00000000a731a9ce x1858043744594880/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771972183 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 3661.223751] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 3661.226674] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3661.230420] LustreError: Skipped 8 previous similar messages [ 3662.358813] LDISKFS-fs (dm-0): recovery complete [ 3662.360442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.425055] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe37ea44e to 0x7f65554fe37f86ee [ 3667.429144] Lustre: Skipped 8 previous similar messages [ 3667.632441] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3667.635781] Lustre: Skipped 10 previous similar messages [ 3668.782782] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3672.888807] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3672.890729] Lustre: Skipped 10 previous similar messages [ 3672.904302] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5547 to 0x0:5569 [ 3672.904860] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5791 to 0x0:5825 [ 3674.842991] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3675.464473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3680.462119] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3681.995268] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 3682.784559] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3682.785464] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3682.787204] LustreError: Skipped 10 previous similar messages [ 3682.791442] Lustre: Skipped 2 previous similar messages [ 3697.561209] LDISKFS-fs (dm-1): recovery complete [ 3697.562752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3699.268515] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3702.776117] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:33 [ 3702.776339] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:198 to 0x280000400:225 [ 3704.632423] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3705.185454] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3710.168789] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3711.669491] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 3712.352383] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 3726.632822] LDISKFS-fs (dm-0): recovery complete [ 3726.634033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.526347] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3731.528449] Lustre: Skipped 7 previous similar messages [ 3732.767145] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3736.817737] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6238 to 0x0:6273 [ 3738.579120] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3739.118592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3743.916641] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3745.440371] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 3760.842875] LDISKFS-fs (dm-1): recovery complete [ 3760.844511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.327109] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3766.257162] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:198 to 0x280000400:257 [ 3766.257257] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:65 [ 3768.053506] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3768.574910] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3787.881098] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 17:31:49 (1771972309) [ 3910.590498] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3921.275484] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 3922.015407] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 3924.961446] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3924.964631] LustreError: Skipped 150 previous similar messages [ 3936.159397] LDISKFS-fs (dm-0): recovery complete [ 3936.160583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3938.272732] LustreError: 189509:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.137@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000eb1fbe23) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3938.280026] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 3938.283633] Lustre: Skipped 27 previous similar messages [ 3938.348859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.353397] Lustre: Skipped 6 previous similar messages [ 3939.636701] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3945.739448] Lustre: lustre-OST0000: deleting orphan objects from 0x0:10266 to 0x0:10305 [ 3945.739792] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5981 to 0x0:6017 [ 3947.601693] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3948.123492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4072.577084] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4083.387117] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 4084.022093] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 4098.831761] LDISKFS-fs (dm-0): recovery complete [ 4098.833124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.424595] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4107.157475] Lustre: lustre-OST0000: deleting orphan objects from 0x0:11696 to 0x0:11713 [ 4107.159240] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7409 to 0x0:7425 [ 4109.051745] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4109.681673] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4122.072156] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 17:37:23 (1771972643) [ 4244.864703] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4255.576238] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 4256.248421] Lustre: Failing over lustre-MDT0000 [ 4256.249758] Lustre: Skipped 5 previous similar messages [ 4256.261058] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 4256.263223] Lustre: Skipped 1 previous similar message [ 4256.442865] Lustre: server umount lustre-MDT0000 complete [ 4256.443994] Lustre: Skipped 5 previous similar messages [ 4258.783900] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.787950] Lustre: Skipped 20 previous similar messages [ 4265.887165] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771972781/real 1771972781] req@000000006ed6ffb7 x1858043753197504/t0(0) o400->MGC192.168.203.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1771972788 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 4265.892529] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4265.894338] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4265.896731] LustreError: Skipped 3 previous similar messages [ 4270.556766] LDISKFS-fs (dm-0): recovery complete [ 4270.558103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4272.096526] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe3b5cba2 to 0x7f65554fe3c6dcdd [ 4272.099040] Lustre: Skipped 3 previous similar messages [ 4272.304391] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4272.306799] Lustre: Skipped 5 previous similar messages [ 4273.382042] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4278.352401] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4278.357023] Lustre: Skipped 5 previous similar messages [ 4278.375182] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7954 to 0x0:7969 [ 4278.375216] Lustre: lustre-OST0000: deleting orphan objects from 0x0:12242 to 0x0:12257 [ 4280.325842] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4280.915246] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4404.954832] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4415.713984] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 4416.386150] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.37@tcp (stopping) [ 4416.991453] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4416.994080] LustreError: Skipped 4 previous similar messages [ 4430.970664] LDISKFS-fs (dm-0): recovery complete [ 4430.972377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4433.965941] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4433.969016] Lustre: Skipped 4 previous similar messages [ 4435.374376] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4440.164203] Lustre: lustre-OST0000: deleting orphan objects from 0x0:12756 to 0x0:12801 [ 4440.165669] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8467 to 0x0:8513 [ 4442.343654] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4442.956367] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4446.676487] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 17:42:48 (1771972968) [ 4569.595315] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4580.376311] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 4581.040374] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4582.367809] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4582.371664] LustreError: Skipped 94 previous similar messages [ 4597.630126] LDISKFS-fs (dm-1): recovery complete [ 4597.633643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.820686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4597.825129] Lustre: Skipped 3 previous similar messages [ 4600.063908] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4602.860424] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 4602.863361] Lustre: Skipped 19 previous similar messages [ 4602.897377] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4048 to 0x2c0000400:4065 [ 4602.897418] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4240 to 0x280000400:4257 [ 4605.798568] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4606.491512] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4732.013916] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4743.118288] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 4744.147862] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4744.151279] Lustre: Skipped 3 previous similar messages [ 4760.266646] LDISKFS-fs (dm-1): recovery complete [ 4760.268222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.311909] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4765.704256] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4240 to 0x280000400:4289 [ 4765.704399] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4048 to 0x2c0000400:4097 [ 4768.562930] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4769.362577] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4773.598266] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 17:48:14 (1771973294) [ 4780.551151] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4782.316744] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 4799.585837] LDISKFS-fs (dm-2): recovery complete [ 4799.587272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4801.610778] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16802 to 0x0:16833 [ 4801.611826] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4240 to 0x280000400:4321 [ 4801.702456] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4806.686067] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4807.400671] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4816.170739] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4817.708921] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 4832.858156] LDISKFS-fs (dm-2): recovery complete [ 4832.859500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4832.901363] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4834.091242] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4834.421411] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4240 to 0x280000400:4353 [ 4834.422505] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16802 to 0x0:16865 [ 4838.149283] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4838.858046] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4844.412130] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 17:49:25 (1771973365) [ 4967.647652] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4971.136708] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4982.000833] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 4982.830339] Lustre: Failing over lustre-MDT0000 [ 4982.832549] Lustre: Skipped 5 previous similar messages [ 4982.959894] Lustre: server umount lustre-MDT0000 complete [ 4982.962449] Lustre: Skipped 5 previous similar messages [ 4984.661879] LustreError: 9972:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973507 with bad export cookie 9179837217077861255 [ 4984.664466] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4984.667854] LustreError: 9972:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4984.674845] LustreError: Skipped 1 previous similar message [ 4994.015192] Lustre: 3338:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771973509/real 1771973509] req@000000007b3d11ac x1858043762390912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1771973516 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 4994.015352] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.028041] Lustre: 3338:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4994.038254] Lustre: Skipped 18 previous similar messages [ 5000.790111] LDISKFS-fs (dm-0): recovery complete [ 5000.791577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.207626] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe3d61b87 to 0x7f65554fe400f7a5 [ 5002.209970] Lustre: Skipped 1 previous similar message [ 5003.792231] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5011.633299] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5011.638572] Lustre: Skipped 5 previous similar messages [ 5019.087406] LDISKFS-fs (dm-1): recovery complete [ 5019.089247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5019.183635] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 5019.187871] LustreError: Skipped 6 previous similar messages [ 5020.938474] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5024.363335] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5024.369354] Lustre: Skipped 5 previous similar messages [ 5024.397097] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4240 to 0x280000400:4385 [ 5024.397157] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4048 to 0x2c0000400:4129 [ 5029.987782] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16802 to 0x0:16897 [ 5029.987885] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12514 to 0x0:12545 [ 5032.551680] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5033.133794] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5033.759572] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5037.841704] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 17:52:39 (1771973559) [ 5041.349376] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5059.113436] LDISKFS-fs (dm-0): recovery complete [ 5059.114728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5068.872322] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5068.876114] Lustre: Skipped 6 previous similar messages [ 5068.977214] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5070.715782] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5075.126050] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 5075.135161] Lustre: 206766:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000009c293ae1 x1858043784410176/t347892391878(347892391878) o101->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:683/0 lens 648/3424 e 0 to 0 dl 1771973603 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 5075.163192] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:16929 [ 5075.163285] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12514 to 0x0:12577 [ 5077.692509] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5078.526126] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5082.820232] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 17:53:24 (1771973604) [ 5085.894834] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5103.488729] LDISKFS-fs (dm-0): recovery complete [ 5103.491325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5113.016693] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5113.018396] LustreError: 206316:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000600fc549 x1858043784410176/t347892391878(347892391878) o101->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:721/0 lens 648/600 e 0 to 0 dl 1771973641 ref 1 fl Interpret:/4/0 rc 301/0 job:'lfs.0' [ 5114.791837] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5119.160796] Lustre: lustre-MDT0000: Client 68a40ec6-d031-4d33-a693-44634e1fc30a (at 192.168.203.37@tcp) reconnected, waiting for 2 clients in recovery for 0:59 [ 5119.165751] Lustre: 206316:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000009742dac1 x1858043784410176/t347892391878(347892391878) o101->68a40ec6-d031-4d33-a693-44634e1fc30a@192.168.203.37@tcp:727/0 lens 648/3424 e 0 to 0 dl 1771973647 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 5119.194943] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:16961 [ 5119.194960] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12579 to 0x0:12609 [ 5121.535767] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5122.073906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5126.321329] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 17:54:07 (1771973647) [ 5143.956372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5147.224953] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5148.218645] Lustre: lustre-MDT0000: Denying connection for new client b11d7275-35c7-4352-a61c-6f9a9a8f4045 (at 192.168.203.37@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5148.228692] Lustre: Skipped 13 previous similar messages [ 5150.707855] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12579 to 0x0:12641 [ 5155.177565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5156.345427] Lustre: lustre-OST0000: Denying connection for new client b11d7275-35c7-4352-a61c-6f9a9a8f4045 (at 192.168.203.37@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5156.349250] Lustre: Skipped 1 previous similar message [ 5156.398724] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5157.232285] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:16993 [ 5157.233140] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4240 to 0x280000400:4417 [ 5160.097258] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 17:54:41 (1771973681) [ 5160.390954] LustreError: 8220:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@00000000ef9ed446 x1858043762836736/t365072220167(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:13/0 lens 1056/4320 e 0 to 0 dl 1771973688 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 5162.744979] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5178.044142] LDISKFS-fs (dm-0): recovery complete [ 5178.046489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.218716] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5183.996731] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17025 [ 5183.996949] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12673 [ 5185.681330] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5186.433552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5190.720690] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 17:55:12 (1771973712) [ 5191.068642] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5191.070082] Lustre: Skipped 1 previous similar message [ 5193.912003] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5196.495256] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5197.122461] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.37@tcp (stopping) [ 5197.124501] Lustre: Skipped 1 previous similar message [ 5204.449304] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5204.456355] LustreError: Skipped 170 previous similar messages [ 5217.080425] LDISKFS-fs (dm-1): recovery complete [ 5217.081619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5217.175539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5217.177917] Lustre: Skipped 9 previous similar messages [ 5218.688123] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5222.369443] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 5222.371028] Lustre: Skipped 37 previous similar messages [ 5222.384403] Lustre: 206995:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000005b4ab3d2 x1858043787304192/t25769805774(0) o36->b11d7275-35c7-4352-a61c-6f9a9a8f4045@192.168.203.37@tcp:104/0 lens 560/448 e 0 to 0 dl 1771973779 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 5222.389868] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4428 to 0x280000400:4449 [ 5222.389868] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4140 to 0x2c0000400:4161 [ 5224.054563] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5224.525341] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5227.923313] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 17:55:49 (1771973749) [ 5228.228972] LustreError: 8221:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@000000009b09ee69 x1858043762868160/t369367187488(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:81/0 lens 2200/4320 e 0 to 0 dl 1771973756 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 5228.234765] LustreError: 8221:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message [ 5230.504368] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5232.902785] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5247.837789] LDISKFS-fs (dm-0): recovery complete [ 5247.839021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.358948] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5253.140785] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17057 [ 5253.140808] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12705 [ 5254.914496] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5255.434769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5273.248037] LDISKFS-fs (dm-1): recovery complete [ 5273.250057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5275.007515] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5278.729167] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4460 to 0x280000400:4481 [ 5278.729175] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4172 to 0x2c0000400:4193 [ 5280.951774] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5281.558418] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5286.049995] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 17:56:47 (1771973807) [ 5292.193695] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5292.512795] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 5294.513568] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5297.150991] LustreError: 158657:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771973819 with bad export cookie 9179837217080745248 [ 5297.157814] LustreError: 158657:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5312.712032] LDISKFS-fs (dm-0): recovery complete [ 5312.714681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5314.907021] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5329.808160] LDISKFS-fs (dm-1): recovery complete [ 5329.810995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5331.546249] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5335.108707] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4492 to 0x280000400:4513 [ 5335.108720] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4204 to 0x2c0000400:4225 [ 5335.166952] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17089 [ 5335.167048] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12737 [ 5337.588268] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5338.156110] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5338.706411] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5342.724813] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 17:57:44 (1771973864) [ 5343.242842] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5343.247492] Lustre: Skipped 2 previous similar messages [ 5343.251558] LustreError: 224770:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@000000001efb54c2 x1858043787356544/t38654705732(0) o36->b11d7275-35c7-4352-a61c-6f9a9a8f4045@192.168.203.37@tcp:196/0 lens 560/448 e 0 to 0 dl 1771973871 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 5343.268695] LustreError: 224770:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message [ 5349.740045] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5350.576652] Lustre: lustre-MDT0001: Client b11d7275-35c7-4352-a61c-6f9a9a8f4045 (at 192.168.203.37@tcp) reconnecting [ 5350.579627] Lustre: Skipped 2 previous similar messages [ 5350.583242] Lustre: 225223:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000000a619f4e x1858043787356544/t38654705732(0) o36->b11d7275-35c7-4352-a61c-6f9a9a8f4045@192.168.203.37@tcp:204/0 lens 560/448 e 0 to 0 dl 1771973879 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 5366.191799] LDISKFS-fs (dm-0): recovery complete [ 5366.193269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.751271] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5372.947465] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12769 [ 5372.947479] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17121 [ 5375.496816] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5376.282345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5380.361084] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 17:58:21 (1771973901) [ 5383.786339] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5399.643743] LDISKFS-fs (dm-1): recovery complete [ 5399.645691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5401.538807] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5405.170663] Lustre: 224770:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000062443833 x1858043787372160/t38654705802(0) o36->b11d7275-35c7-4352-a61c-6f9a9a8f4045@192.168.203.37@tcp:258/0 lens 560/448 e 0 to 0 dl 1771973933 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 5405.178403] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4534 to 0x280000400:4577 [ 5405.178420] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4246 to 0x2c0000400:4289 [ 5407.616917] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5408.375907] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5412.505638] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 17:58:54 (1771973934) [ 5418.825829] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5420.208886] Lustre: lustre-MDT0001: Client b11d7275-35c7-4352-a61c-6f9a9a8f4045 (at 192.168.203.37@tcp) reconnecting [ 5422.009104] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5438.888541] LDISKFS-fs (dm-0): recovery complete [ 5438.890625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5440.733799] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5444.119444] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12801 [ 5444.119602] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17153 [ 5446.682046] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5447.478463] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5465.878643] LDISKFS-fs (dm-1): recovery complete [ 5465.880379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.858412] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5471.232317] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4588 to 0x280000400:4609 [ 5471.232405] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4300 to 0x2c0000400:4321 [ 5473.335052] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5474.143973] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5478.480246] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 17:59:59 (1771973999) [ 5478.907850] LustreError: 224769:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@000000001e14b11f x1858043787407104/t47244640326(0) o36->b11d7275-35c7-4352-a61c-6f9a9a8f4045@192.168.203.37@tcp:332/0 lens 560/448 e 0 to 0 dl 1771974007 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 5478.920383] LustreError: 224769:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 2 previous similar messages [ 5485.117504] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5486.257244] Lustre: lustre-MDT0001: Client b11d7275-35c7-4352-a61c-6f9a9a8f4045 (at 192.168.203.37@tcp) reconnecting [ 5486.264735] Lustre: 224769:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000017c30b44 x1858043787407104/t47244640326(0) o36->b11d7275-35c7-4352-a61c-6f9a9a8f4045@192.168.203.37@tcp:339/0 lens 560/448 e 0 to 0 dl 1771974014 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 5486.277106] Lustre: 224769:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5488.562481] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5491.316929] LustreError: 158653:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974013 with bad export cookie 9179837217080758912 [ 5491.324827] LustreError: 158653:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5507.317381] LDISKFS-fs (dm-0): recovery complete [ 5507.319992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.981639] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5526.303423] LDISKFS-fs (dm-1): recovery complete [ 5526.307481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5528.122285] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5531.832466] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17185 [ 5531.832466] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12833 [ 5531.840201] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4332 to 0x2c0000400:4353 [ 5531.840230] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4620 to 0x280000400:4641 [ 5533.700903] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5534.181871] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5534.633601] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5537.962017] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 18:00:59 (1771974059) [ 5540.485673] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5546.976283] LustreError: 239018:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000828e7501 x1858043763008896/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/0/ffffffff rc 0/-1 job:'umount.0' [ 5546.980790] LustreError: 239018:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5561.038750] LDISKFS-fs (dm-1): recovery complete [ 5561.040182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5562.305596] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5566.452714] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4673 [ 5566.452714] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4364 to 0x2c0000400:4385 [ 5568.109131] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5568.635937] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5571.888776] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 18:01:33 (1771974093) [ 5574.358664] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5589.104845] LDISKFS-fs (dm-0): recovery complete [ 5589.106234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.349513] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5595.131803] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17217 [ 5595.131823] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12865 [ 5597.610770] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5598.231323] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5601.995301] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 18:02:03 (1771974123) [ 5602.493742] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5602.497088] Lustre: Skipped 5 previous similar messages [ 5605.589840] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5609.060604] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5609.885661] Lustre: Failing over lustre-MDT0000 [ 5609.887807] Lustre: Skipped 19 previous similar messages [ 5610.262121] Lustre: server umount lustre-MDT0000 complete [ 5610.263969] Lustre: Skipped 19 previous similar messages [ 5610.466121] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5610.473556] Lustre: Skipped 60 previous similar messages [ 5613.535216] Lustre: 239691:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771974125/real 1771974125] req@000000007756afbd x1858043763035392/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1488/4320 e 0 to 1 dl 1771974136 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'osp_up0-1.0' [ 5613.551467] Lustre: 239691:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 51 previous similar messages [ 5616.607194] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5616.610753] LustreError: Skipped 10 previous similar messages [ 5625.516800] LDISKFS-fs (dm-0): recovery complete [ 5625.519097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.016771] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe402747d to 0x7f65554fe4027ac0 [ 5634.022560] Lustre: Skipped 10 previous similar messages [ 5634.736744] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5634.741305] Lustre: Skipped 19 previous similar messages [ 5635.739582] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5639.172479] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5639.178591] Lustre: Skipped 19 previous similar messages [ 5639.204978] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17249 [ 5639.205127] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12897 [ 5641.926694] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5642.664753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5649.377195] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5649.382081] LustreError: Skipped 15 previous similar messages [ 5660.160404] LDISKFS-fs (dm-1): recovery complete [ 5660.162528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5662.197482] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5665.793622] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4705 [ 5665.793678] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4364 to 0x2c0000400:4417 [ 5668.035959] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5668.605391] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5672.870497] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 18:03:14 (1771974194) [ 5676.991494] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5679.515356] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5679.583837] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 5681.909910] LustreError: 6221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974204 with bad export cookie 9179837217080769216 [ 5681.917076] LustreError: 6221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5698.349257] LDISKFS-fs (dm-0): recovery complete [ 5698.352057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.730525] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5698.738995] Lustre: Skipped 19 previous similar messages [ 5700.927396] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5717.218334] LDISKFS-fs (dm-1): recovery complete [ 5717.221123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5719.464736] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5723.368726] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17281 [ 5723.369135] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12929 [ 5723.389112] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4737 [ 5723.389139] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4364 to 0x2c0000400:4449 [ 5725.508871] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5726.032281] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5726.503165] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5729.725137] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 18:04:11 (1771974251) [ 5732.472458] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5737.136600] Lustre: lustre-MDT0001: Client b11d7275-35c7-4352-a61c-6f9a9a8f4045 (at 192.168.203.37@tcp) reconnecting [ 5737.140358] Lustre: 248349:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000b4a875b4 x1858043787455424/t64424509442(0) o36->b11d7275-35c7-4352-a61c-6f9a9a8f4045@192.168.203.37@tcp:590/0 lens 496/2888 e 0 to 0 dl 1771974265 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0' [ 5737.145387] Lustre: 248349:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5746.871678] LDISKFS-fs (dm-0): recovery complete [ 5746.872834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5752.597146] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5756.410071] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12961 [ 5756.410072] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17313 [ 5758.113833] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5758.703616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5762.028100] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 18:04:43 (1771974283) [ 5762.394430] LustreError: 249037:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply req@000000001e0922f9 x1858043787461952/t64424509444(0) o36->b11d7275-35c7-4352-a61c-6f9a9a8f4045@192.168.203.37@tcp:615/0 lens 496/456 e 0 to 0 dl 1771974290 ref 1 fl Interpret:/0/0 rc 0/0 job:'rmdir.0' [ 5762.399670] LustreError: 249037:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 5 previous similar messages [ 5764.723445] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5779.348196] LDISKFS-fs (dm-1): recovery complete [ 5779.349535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5780.781867] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5784.564542] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4769 [ 5784.564573] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4364 to 0x2c0000400:4481 [ 5786.277658] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5786.797475] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5790.108356] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 18:05:11 (1771974311) [ 5792.636333] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5794.853533] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5805.025139] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5805.029080] LustreError: Skipped 295 previous similar messages [ 5809.509926] LDISKFS-fs (dm-0): recovery complete [ 5809.511114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5813.216812] LustreError: 256071:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.137@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000dea73abd) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5814.554213] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5818.392103] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:12993 [ 5818.392150] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17345 [ 5820.996932] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5821.783559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5839.751610] LDISKFS-fs (dm-1): recovery complete [ 5839.754366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.886726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5839.889943] Lustre: Skipped 19 previous similar messages [ 5841.341604] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5844.962703] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 5844.966469] Lustre: Skipped 78 previous similar messages [ 5844.997305] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4801 [ 5844.997328] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4364 to 0x2c0000400:4513 [ 5847.061465] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5847.796411] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5851.855229] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 18:06:13 (1771974373) [ 5855.787544] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5859.274844] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5860.319927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5860.321857] Lustre: Skipped 18 previous similar messages [ 5862.067095] LustreError: 158657:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974384 with bad export cookie 9179837217080776734 [ 5862.074434] LustreError: 158657:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5878.307749] LDISKFS-fs (dm-0): recovery complete [ 5878.310305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5881.944350] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5897.865326] LDISKFS-fs (dm-1): recovery complete [ 5897.868481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5899.906660] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5903.646191] Lustre: lustre-OST0001: deleting orphan objects from 0x0:12643 to 0x0:13025 [ 5903.646191] Lustre: lustre-OST0000: deleting orphan objects from 0x0:16899 to 0x0:17377 [ 5903.651932] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4833 [ 5903.653464] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4364 to 0x2c0000400:4545 [ 5906.296551] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5907.065578] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5907.777574] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5911.009145] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 18:07:12 (1771974432) [ 5911.762637] Lustre: 262195:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b11d7275-35c7-4352-a61c-6f9a9a8f4045 at adminstrative request [ 5917.225708] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 18:07:18 (1771974438) [ 5934.781282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5938.429484] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5942.285189] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17428 to 0x0:17473 [ 5942.285218] Lustre: lustre-OST0001: deleting orphan objects from 0x0:13077 to 0x0:13121 [ 5943.866104] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5944.334725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5948.650567] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 18:07:49 (1771974469) [ 5967.267129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.968577] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17574 to 0x0:17601 [ 5968.970857] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4865 [ 5969.477251] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5974.714378] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5975.553857] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5980.582399] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 18:08:21 (1771974501) [ 5986.092469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5988.144264] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5989.189519] Lustre: lustre-MDT0000: Denying connection for new client 6680bdb5-5d75-48bc-b8df-4e5591a5c1e0 (at 192.168.203.37@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5991.417431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17574 to 0x0:17633 [ 5991.417460] Lustre: lustre-OST0001: deleting orphan objects from 0x0:13077 to 0x0:13153 [ 5997.600463] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 18:08:38 (1771974518) [ 6001.445456] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6018.165826] LDISKFS-fs (dm-2): recovery complete [ 6018.168447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6019.878921] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6020.332503] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17635 to 0x0:17665 [ 6020.333331] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4897 [ 6024.406595] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6025.060810] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6029.353631] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 18:09:10 (1771974550) [ 6032.498283] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6050.411334] LDISKFS-fs (dm-2): recovery complete [ 6050.414097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6052.056193] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.37@tcp inode [0x200030971:0x5:0x0] object 0x0:17666 extent [0-4194303]: client csum 2f9bdd15, server csum 1d7de9e5 [ 6052.107850] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17667 to 0x0:17697 [ 6052.108952] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4929 [ 6052.208806] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6056.547346] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6057.283944] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6061.396858] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 18:09:42 (1771974582) [ 6064.900498] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6067.607530] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6072.498464] LustreError: 158654:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771974595 with bad export cookie 9179837217080802802 [ 6072.505759] LustreError: 158654:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6088.433470] LDISKFS-fs (dm-0): recovery complete [ 6088.436164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6091.697129] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6095.016553] Lustre: lustre-OST0001: deleting orphan objects from 0x0:13077 to 0x0:13185 [ 6107.989209] LDISKFS-fs (dm-2): recovery complete [ 6107.992072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6109.838447] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4961 [ 6110.076693] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6117.500900] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 18:10:38 (1771974638) [ 6139.855349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.031916] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6147.084396] Lustre: lustre-OST0001: deleting orphan objects from 0x0:13077 to 0x0:13217 [ 6149.740578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6151.813506] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6152.818232] Lustre: lustre-OST0000: Denying connection for new client 0a28038f-6634-4886-9de7-6159e36b26d9 (at 192.168.203.37@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 6152.829070] Lustre: Skipped 1 previous similar message [ 6158.000684] Lustre: lustre-OST0000: Denying connection for new client 0a28038f-6634-4886-9de7-6159e36b26d9 (at 192.168.203.37@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 6158.004858] Lustre: Skipped 1 previous similar message [ 6168.241233] Lustre: lustre-OST0000: Denying connection for new client 0a28038f-6634-4886-9de7-6159e36b26d9 (at 192.168.203.37@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 6168.251215] Lustre: Skipped 1 previous similar message [ 6188.720785] Lustre: lustre-OST0000: Denying connection for new client 0a28038f-6634-4886-9de7-6159e36b26d9 (at 192.168.203.37@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 6188.726354] Lustre: Skipped 3 previous similar messages [ 6221.000240] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6221.005266] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6221.017204] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17748 to 0x0:17769 [ 6221.018427] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:4993 [ 6225.587366] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec [ 6231.414477] Lustre: DEBUG MARKER: free_before: 7646580 free_after: 7646580 [ 6234.153446] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 18:12:35 (1771974755) [ 6235.548219] Lustre: Failing over lustre-OST0001 [ 6235.549281] Lustre: Skipped 18 previous similar messages [ 6235.562270] Lustre: server umount lustre-OST0001 complete [ 6235.563468] Lustre: Skipped 18 previous similar messages [ 6236.639688] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.647373] Lustre: Skipped 52 previous similar messages [ 6249.544265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6250.161535] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6250.166418] Lustre: Skipped 18 previous similar messages [ 6251.330556] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6251.335098] Lustre: Skipped 18 previous similar messages [ 6251.340165] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4364 to 0x2c0000400:4577 [ 6251.341815] Lustre: lustre-OST0001: deleting orphan objects from 0x0:13220 to 0x0:13249 [ 6251.735133] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6256.579915] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 18:12:57 (1771974777) [ 6272.199517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6274.169079] LustreError: 278795:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6274.174258] LustreError: 278795:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages [ 6274.213507] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6274.975206] Lustre: *** cfs_fail_loc=715, val=0*** [ 6280.671266] Lustre: 3337:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1771974796/real 1771974796] req@000000001eec0814 x1858043763277888/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1771974803 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 6280.686449] Lustre: 3337:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 34 previous similar messages [ 6280.691855] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:58 [ 6280.697835] Lustre: Skipped 1 previous similar message [ 6281.695251] Lustre: *** cfs_fail_loc=715, val=0*** [ 6281.697801] Lustre: Skipped 2 previous similar messages [ 6287.839712] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:51 [ 6287.845466] Lustre: Skipped 2 previous similar messages [ 6288.863241] Lustre: *** cfs_fail_loc=715, val=0*** [ 6288.865598] Lustre: Skipped 2 previous similar messages [ 6294.192909] Lustre: lustre-OST0000: Client 0a28038f-6634-4886-9de7-6159e36b26d9 (at 192.168.203.37@tcp) reconnected, waiting for 3 clients in recovery for 0:44 [ 6294.199906] Lustre: Skipped 1 previous similar message [ 6295.263233] Lustre: *** cfs_fail_loc=715, val=0*** [ 6295.265555] Lustre: Skipped 2 previous similar messages [ 6301.360929] Lustre: lustre-OST0000: Client 0a28038f-6634-4886-9de7-6159e36b26d9 (at 192.168.203.37@tcp) reconnected, waiting for 3 clients in recovery for 0:37 [ 6301.367992] Lustre: Skipped 2 previous similar messages [ 6302.431221] Lustre: *** cfs_fail_loc=715, val=0*** [ 6302.433747] Lustre: Skipped 2 previous similar messages [ 6308.528952] Lustre: lustre-OST0000: Client 0a28038f-6634-4886-9de7-6159e36b26d9 (at 192.168.203.37@tcp) reconnected, waiting for 3 clients in recovery for 0:30 [ 6308.533863] Lustre: Skipped 2 previous similar messages [ 6314.215130] LustreError: 278795:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 6314.219848] LustreError: 278795:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 4 previous similar messages [ 6314.232979] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:4652 to 0x280000400:5025 [ 6314.234096] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17773 to 0x0:17801 [ 6316.980604] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6317.730200] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6322.159400] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 18:14:03 (1771974843) [ 6336.991164] LustreError: 166-1: MGC192.168.203.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6336.997180] LustreError: Skipped 8 previous similar messages [ 6338.558130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.138409] Lustre: Evicted from MGS (at 192.168.203.137@tcp) after server handle changed from 0x7f65554fe40326d3 to 0x7f65554fe4033822 [ 6343.142728] Lustre: Skipped 8 previous similar messages [ 6343.243969] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6343.249112] Lustre: Skipped 18 previous similar messages [ 6344.965953] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6345.759175] Lustre: *** cfs_fail_loc=715, val=0*** [ 6345.761753] Lustre: Skipped 5 previous similar messages [ 6348.263588] LustreError: 280356:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6352.048975] Lustre: lustre-MDT0000: Client 0a28038f-6634-4886-9de7-6159e36b26d9 (at 192.168.203.37@tcp) reconnected, waiting for 2 clients in recovery for 0:57 [ 6352.055831] Lustre: Skipped 2 previous similar messages [ 6355.423636] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6361.567642] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6362.591135] Lustre: *** cfs_fail_loc=715, val=0*** [ 6362.593607] Lustre: Skipped 4 previous similar messages [ 6368.735743] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6373.552749] Lustre: lustre-MDT0000: Client 0a28038f-6634-4886-9de7-6159e36b26d9 (at 192.168.203.37@tcp) reconnected, waiting for 2 clients in recovery for 0:36 [ 6373.559860] Lustre: Skipped 2 previous similar messages [ 6375.903675] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6383.071581] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6396.063231] Lustre: *** cfs_fail_loc=715, val=0*** [ 6396.065489] Lustre: Skipped 8 previous similar messages [ 6397.407542] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6397.411951] Lustre: Skipped 1 previous similar message [ 6409.393070] Lustre: lustre-MDT0000: Recovery already passed deadline 0:00. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 6416.560707] Lustre: lustre-MDT0000: Recovery already passed deadline 0:06. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 6417.887439] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6417.892266] Lustre: Skipped 2 previous similar messages [ 6423.728984] Lustre: lustre-MDT0000: Recovery already passed deadline 0:13. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [ 6428.351121] LustreError: 280356:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 6428.360167] Lustre: 280356:0:(ldlm_lib.c:2829:target_recovery_thread()) too long recovery - read logs [ 6428.364998] LustreError: dumping log to /tmp/lustre-log.1771974950.280356 [ 6428.420417] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17812 to 0x0:17833 [ 6428.420479] Lustre: lustre-OST0001: deleting orphan objects from 0x0:13260 to 0x0:13281 [ 6431.290869] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6432.100800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6436.644906] Lustre: DEBUG MARKER: == replay-single test complete, duration 6270 sec ======== 18:15:57 (1771974957) [ 6440.417767] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.425098] LustreError: Skipped 235 previous similar messages [ 6457.147202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.970516] Lustre: MGC192.168.203.137@tcp: Connection restored to (at 0@lo) [ 6463.973911] Lustre: Skipped 46 previous similar messages [ 6464.121744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6464.124358] Lustre: Skipped 14 previous similar messages [ 6466.142593] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6469.635376] Lustre: lustre-OST0000: deleting orphan objects from 0x0:17812 to 0x0:17865 [ 6469.635407] Lustre: lustre-OST0001: deleting orphan objects from 0x0:13260 to 0x0:13313 [ 6472.263683] Lustre: DEBUG MARKER: oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6472.916067] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6479.841074] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6479.841660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6479.845624] LustreError: Skipped 21 previous similar messages [ 6479.848556] Lustre: Skipped 3 previous similar messages [ 6485.771899] LustreError: 7242:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1771975008 with bad export cookie 9179837217080835849 [ 6485.778522] LustreError: 7242:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6508.743691] Lustre: DEBUG MARKER: oleg337-server.virtnet: executing unload_modules_local [ 6510.322948] Key type lgssc unregistered [ 6510.495533] LNet: 285189:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6510.501133] LNet: Removed LNI 192.168.203.137@tcp [ 6511.026823] Key type .llcrypt unregistered [ 6511.029101] Key type ._llcrypt unregistered