[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 443717711 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002392] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006957] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009142] LSM: Security Framework initializing [ 0.010049] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.012064] *** VALIDATE selinux *** [ 0.020434] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026071] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027111] *** VALIDATE tmpfs *** [ 0.029222] *** VALIDATE proc *** [ 0.030074] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.032270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034119] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036028] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.039943] debug: unmapping init [mem 0xffffffffb5859000-0xffffffffb5860fff] [ 0.041163] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043023] ... version: 2 [ 0.043884] ... bit width: 48 [ 0.044010] ... generic registers: 4 [ 0.044755] ... value mask: 0000ffffffffffff [ 0.045014] ... max period: 00007fffffffffff [ 0.046010] ... fixed-purpose events: 3 [ 0.046830] ... event mask: 000000070000000f [ 0.047260] rcu: Hierarchical SRCU implementation. [ 0.049203] smp: Bringing up secondary CPUs ... [ 0.050412] x86: Booting SMP configuration: [ 0.051023] .... node #0, CPUs: #1 #2 #3 [ 0.062229] smp: Brought up 1 node, 4 CPUs [ 0.064028] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142020] node 0 deferred pages initialised in 73ms [ 0.145309] devtmpfs: initialized [ 0.146314] x86/mm: Memory block size: 128MB [ 0.149998] gcov: version magic: 0x41383552 [ 0.152314] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158333] pinctrl core: initialized pinctrl subsystem [ 0.160176] [ 0.160782] ************************************************************* [ 0.163012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166012] ** ** [ 0.168011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171014] ** ** [ 0.173011] ** This means that this kernel is built to expose internal ** [ 0.175011] ** IOMMU data structures, which may compromise security on ** [ 0.177011] ** your system. ** [ 0.180011] ** ** [ 0.182009] ** If you see this message and you are not debugging the ** [ 0.184013] ** kernel, report this immediately to your vendor! ** [ 0.186010] ** ** [ 0.189012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.191011] ************************************************************* [ 0.194574] NET: Registered protocol family 16 [ 0.196579] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.199057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.202064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.206103] cpuidle: using governor menu [ 0.207786] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210445] PCI: Using configuration type 1 for base access [ 0.212121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.222101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.223073] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225049] cryptd: max_cpu_qlen set to 1000 [ 0.227255] ACPI: Added _OSI(Module Device) [ 0.228037] ACPI: Added _OSI(Processor Device) [ 0.229000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231028] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235426] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241535] ACPI: Interpreter enabled [ 0.242049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244010] ACPI: Using IOAPIC for interrupt routing [ 0.246105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260545] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266058] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268989] acpiphp: Slot [2] registered [ 0.270069] acpiphp: Slot [5] registered [ 0.271086] acpiphp: Slot [6] registered [ 0.272097] acpiphp: Slot [7] registered [ 0.274094] acpiphp: Slot [8] registered [ 0.275098] acpiphp: Slot [9] registered [ 0.277116] acpiphp: Slot [10] registered [ 0.278000] acpiphp: Slot [3] registered [ 0.279159] acpiphp: Slot [4] registered [ 0.281089] acpiphp: Slot [11] registered [ 0.283075] acpiphp: Slot [12] registered [ 0.284070] acpiphp: Slot [13] registered [ 0.285046] acpiphp: Slot [14] registered [ 0.287068] acpiphp: Slot [15] registered [ 0.288051] acpiphp: Slot [16] registered [ 0.289217] acpiphp: Slot [17] registered [ 0.291092] acpiphp: Slot [18] registered [ 0.292023] acpiphp: Slot [19] registered [ 0.293052] acpiphp: Slot [20] registered [ 0.294086] acpiphp: Slot [21] registered [ 0.295090] acpiphp: Slot [22] registered [ 0.297081] acpiphp: Slot [23] registered [ 0.298106] acpiphp: Slot [24] registered [ 0.300105] acpiphp: Slot [25] registered [ 0.302124] acpiphp: Slot [26] registered [ 0.303095] acpiphp: Slot [27] registered [ 0.305127] acpiphp: Slot [28] registered [ 0.307092] acpiphp: Slot [29] registered [ 0.308073] acpiphp: Slot [30] registered [ 0.309063] acpiphp: Slot [31] registered [ 0.310055] PCI host bridge to bus 0000:00 [ 0.312019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.314016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.319016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.321019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.322235] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.323901] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327005] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.335396] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.339044] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.341016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.343012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.345013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.347498] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349612] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351024] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.352886] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.358013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.368839] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.373018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.380332] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.388020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.394021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.410013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.419434] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.425013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.429013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.439018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.445000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.449013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.453012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.463014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.473627] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.483015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.487016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.497016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.502807] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.505995] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.510015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.519018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.524000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.529012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.533019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.547017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.555069] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.557302] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.559362] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.561427] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.562157] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.566162] iommu: Default domain type: Passthrough [ 0.568526] SCSI subsystem initialized [ 0.570114] ACPI: bus type USB registered [ 0.571111] usbcore: registered new interface driver usbfs [ 0.573067] usbcore: registered new interface driver hub [ 0.575095] usbcore: registered new device driver usb [ 0.577174] pps_core: LinuxPPS API ver. 1 registered [ 0.579012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.583062] PTP clock support registered [ 0.585048] EDAC MC: Ver: 3.0.0 [ 0.587127] PCI: Using ACPI for IRQ routing [ 0.588477] NetLabel: Initializing [ 0.590013] NetLabel: domain hash size = 128 [ 0.591009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.594080] NetLabel: unlabeled traffic allowed by default [ 0.596128] vgaarb: loaded [ 0.597333] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.599010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.605913] clocksource: Switched to clocksource kvm-clock [ 0.698657] VFS: Disk quotas dquot_6.6.0 [ 0.699717] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.701173] *** VALIDATE ramfs *** [ 0.701970] *** VALIDATE hugetlbfs *** [ 0.703128] pnp: PnP ACPI init [ 0.704748] pnp: PnP ACPI: found 6 devices [ 0.718035] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.720270] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.721772] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.723247] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.724779] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.726548] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.729107] NET: Registered protocol family 2 [ 0.731740] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.735473] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.738246] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.742390] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.744925] TCP: Hash tables configured (established 65536 bind 65536) [ 0.747324] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.749566] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.751586] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.753571] NET: Registered protocol family 1 [ 0.755329] RPC: Registered named UNIX socket transport module. [ 0.756836] RPC: Registered udp transport module. [ 0.758082] RPC: Registered tcp transport module. [ 0.759232] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.760929] NET: Registered protocol family 44 [ 0.761805] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.763488] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.764741] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.766449] PCI: CLS 0 bytes, default 64 [ 0.768560] Unpacking initramfs... [ 2.130389] debug: unmapping init [mem 0xffff98abfcc54000-0xffff98abfffbffff] [ 2.134166] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.136563] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.139334] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.651186] Initialise system trusted keyrings [ 2.652684] Key type blacklist registered [ 2.654082] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.663497] zbud: loaded [ 2.666525] *** VALIDATE nfs *** [ 2.667806] *** VALIDATE nfs4 *** [ 2.669355] pstore: using deflate compression [ 2.673238] Platform Keyring initialized [ 2.779550] NET: Registered protocol family 38 [ 2.781296] Key type asymmetric registered [ 2.782703] Asymmetric key parser 'x509' registered [ 2.784455] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.786225] io scheduler mq-deadline registered [ 2.787405] io scheduler kyber registered [ 2.788551] io scheduler bfq registered [ 2.790070] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.792087] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.794627] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.796359] ACPI: Power Button [PWRF] [ 2.800040] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.805591] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.818566] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.823126] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.834563] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.862340] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.889492] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.894334] Non-volatile memory driver v1.3 [ 2.895912] Linux agpgart interface v0.103 [ 2.932062] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 2.934604] vda: detected capacity change from 0 to 34836480 [ 2.955144] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.959266] vdb: detected capacity change from 0 to 1073741824 [ 2.982727] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.985799] vdc: detected capacity change from 0 to 2621440000 [ 2.999793] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.002593] vdd: detected capacity change from 0 to 2621440000 [ 3.016775] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.018971] vde: detected capacity change from 0 to 4294967296 [ 3.031587] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.034020] vdf: detected capacity change from 0 to 4294967296 [ 3.041366] libphy: Fixed MDIO Bus: probed [ 3.079654] usbcore: registered new interface driver usbserial_generic [ 3.082392] usbserial: USB Serial support registered for generic [ 3.084829] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.088477] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.090222] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.092663] mousedev: PS/2 mouse device common for all mice [ 3.095667] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.097598] rtc_cmos 00:05: RTC can wake from S4 [ 3.101770] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.104418] rtc_cmos 00:05: registered as rtc0 [ 3.108057] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.108597] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.110567] intel_pstate: CPU model not supported [ 3.115923] hid: raw HID events driver (C) Jiri Kosina [ 3.117410] usbcore: registered new interface driver usbhid [ 3.119625] usbhid: USB HID core driver [ 3.121196] drop_monitor: Initializing network drop monitor service [ 3.123023] Initializing XFRM netlink socket [ 3.124731] NET: Registered protocol family 10 [ 3.127922] Segment Routing with IPv6 [ 3.129341] NET: Registered protocol family 17 [ 3.131432] mpls_gso: MPLS GSO support [ 3.136811] RAS: Correctable Errors collector initialized. [ 3.139160] AVX version of gcm_enc/dec engaged. [ 3.140974] AES CTR mode by8 optimization enabled [ 3.214964] sched_clock: Marking stable (3214941376, 0)->(4017949274, -803007898) [ 3.218418] registered taskstats version 1 [ 3.220677] Loading compiled-in X.509 certificates [ 3.223980] zswap: loaded using pool lzo/zbud [ 3.249189] Key type big_key registered [ 3.258829] Key type encrypted registered [ 3.259865] ima: No TPM chip found, activating TPM-bypass! [ 3.261218] ima: Allocated hash algorithm: sha1 [ 3.262254] ima: No architecture policies found [ 3.263353] evm: Initialising EVM extended attributes: [ 3.264441] evm: security.selinux [ 3.265238] evm: security.ima [ 3.266387] evm: security.capability [ 3.267403] evm: HMAC attrs: 0x1 [ 3.268903] rtc_cmos 00:05: setting system clock to 2026-04-14 15:12:40 UTC (1776179560) [ 3.272969] debug: unmapping init [mem 0xffffffffb6803000-0xffffffffb69fffff] [ 3.275015] debug: unmapping init [mem 0xffffffffb5582000-0xffffffffb5858fff] [ 3.285285] Write protecting the kernel read-only data: 28672k [ 3.289882] debug: unmapping init [mem 0xffffffffb3c03000-0xffffffffb3dfffff] [ 3.292840] debug: unmapping init [mem 0xffffffffb4514000-0xffffffffb45fffff] [ 3.326858] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.332903] systemd[1]: Detected virtualization kvm. [ 3.334230] systemd[1]: Detected architecture x86-64. [ 3.335494] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.361969] systemd[1]: No hostname configured. [ 3.363182] systemd[1]: Set hostname to . [ 3.365179] random: systemd: uninitialized urandom read (16 bytes read) [ 3.367524] systemd[1]: Initializing machine ID from random generator. [ 3.434290] random: ln: uninitialized urandom read (6 bytes read) [ 3.503258] random: systemd: uninitialized urandom read (16 bytes read) [ 3.505126] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.508644] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.511857] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.056054] device-mapper: uevent: version 1.0.3 [ 4.058665] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [[ 4.811343] random: fast init done  OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.934882] virtio_net virtio0 ens2: renamed from eth0 [ 4.995943] scsi host0: ata_piix [ 5.011211] scsi host1: ata_piix [ 5.012756] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.014543] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.637456] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.771852] random: crng init done [ 9.773212] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.181430] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.358444] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.584430] SELinux: Disabled at runtime. [ 11.645530] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.659117] systemd[1]: Detected virtualization kvm. [ 11.662471] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.331239] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.335373] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.341390] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.347569] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.350135] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.360803] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.365388] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ 12.443140] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.898312] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.169242] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.209633] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.351465] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.364456] EDAC sbridge: Ver: 1.1.2 [ 15.091460] Key type dns_resolver registered [ 15.404742] NFS: Registering the id_resolver key type [ 15.406159] Key type id_resolver registered [ 15.407302] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg222-server login: [ 54.182012] hrtimer: interrupt took 4551367 ns [ 59.747779] libcfs: loading out-of-tree module taints kernel. [ 59.795320] alg: No test for adler32 (adler32-zlib) [ 60.550243] Key type ._llcrypt registered [ 60.553447] Key type .llcrypt registered [ 60.736837] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_hostid [ 77.226338] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 78.647348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.157926] Lustre: Lustre: Build Version: 2.15.8_13_g05ba8a9 [ 79.856314] LNet: Added LNI 192.168.202.122@tcp [8/256/0/180] [ 79.859460] LNet: Accept secure, port 988 [ 81.552127] Key type lgssc registered [ 82.670648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.274086] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.114178] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.560069] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.726685] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 119.973451] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 134.653903] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local [ 147.887691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 147.954388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 149.144255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 149.176077] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 149.289628] Lustre: lustre-MDT0000: new disk, initializing [ 149.372769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.395149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 153.587802] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 165.872869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 165.962316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.081614] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 166.138068] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 166.143655] Lustre: Skipped 1 previous similar message [ 166.249269] Lustre: lustre-MDT0001: new disk, initializing [ 166.361708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 166.405655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 166.421085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 170.836731] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 182.556813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.600466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 182.603474] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 182.824354] Lustre: lustre-OST0000: new disk, initializing [ 182.826970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 182.861490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 187.243574] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 190.492875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 190.502167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.568652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.635384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.642612] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 200.754541] Lustre: lustre-OST0001: new disk, initializing [ 200.760034] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 200.808743] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.733545] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 207.380255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 207.391196] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.294841] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.039038] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 233.844891] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing check_logdir /tmp/testlogs/ [ 237.753502] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing yml_node [ 244.016292] Lustre: DEBUG MARKER: Client: 2.15.8.13 [ 246.031598] Lustre: DEBUG MARKER: MDS: 2.15.8.13 [ 248.218831] Lustre: DEBUG MARKER: OSS: 2.15.8.13 [ 249.702410] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Apr 14 11:16:46 EDT 2026 [ 258.321335] Lustre: DEBUG MARKER: excepting tests: 59 [ 264.545798] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing check_config_client /mnt/lustre [ 283.250042] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 286.481557] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 290.227584] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 293.099985] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 11:17:29 (1776179849) [ 299.824851] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 301.793380] Lustre: Failing over lustre-MDT0000 [ 301.909911] Lustre: server umount lustre-MDT0000 complete [ 302.223449] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.611190] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 304.618888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.638112] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.320515] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.331947] LustreError: Skipped 3 previous similar messages [ 309.729662] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.740483] LustreError: Skipped 3 previous similar messages [ 314.850992] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.858927] LustreError: Skipped 2 previous similar messages [ 322.834900] LDISKFS-fs (dm-0): recovery complete [ 322.837358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 322.858616] LustreError: 11-0: MGC192.168.202.122@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 322.863553] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.865680] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.879681] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba134e505 to 0x180b24bba134ebb1 [ 322.880199] LustreError: Skipped 12 previous similar messages [ 322.887596] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 323.072799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.492880] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 327.802680] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 328.170855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 328.238588] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 334.339561] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 335.775511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 342.672871] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 11:18:19 (1776179899) [ 344.337726] Lustre: Failing over lustre-OST0000 [ 344.387588] Lustre: server umount lustre-OST0000 complete [ 344.545234] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 344.557798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.573924] Lustre: Skipped 3 previous similar messages [ 344.576936] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.643585] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 348.660535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.473256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 362.611549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 363.650141] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 364.278982] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 364.279841] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.202.122@tcp (at 0@lo) [ 364.298129] Lustre: Skipped 3 previous similar messages [ 364.312243] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 366.033661] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 375.635642] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 377.371748] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 386.421289] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 11:19:02 (1776179942) [ 393.919849] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 395.488315] Lustre: Failing over lustre-MDT0000 [ 395.664852] Lustre: server umount lustre-MDT0000 complete [ 398.817634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.821519] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.834407] Lustre: Skipped 3 previous similar messages [ 398.857304] LustreError: Skipped 12 previous similar messages [ 405.840183] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179956/real 1776179956] req@0000000078d83342 x1862459339126080/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179963 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 405.869965] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 421.682777] LDISKFS-fs (dm-0): recovery complete [ 421.697092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.399700] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba134ebb1 to 0x180b24bba134f5eb [ 423.410829] Lustre: MGC192.168.202.122@tcp: Connection restored to 192.168.202.122@tcp (at 0@lo) [ 423.427245] Lustre: Skipped 1 previous similar message [ 423.998740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.758361] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 429.034706] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 429.046409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 431.124638] Lustre: lustre-MDT0000: Denying connection for new client c6ccb568-5d68-4ea3-9e65-665cc299120d (at 192.168.202.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:39 [ 436.343694] Lustre: lustre-MDT0000: Denying connection for new client c6ccb568-5d68-4ea3-9e65-665cc299120d (at 192.168.202.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:34 [ 441.466341] Lustre: lustre-MDT0000: Denying connection for new client c6ccb568-5d68-4ea3-9e65-665cc299120d (at 192.168.202.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:29 [ 446.579434] Lustre: lustre-MDT0000: Denying connection for new client c6ccb568-5d68-4ea3-9e65-665cc299120d (at 192.168.202.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:24 [ 451.713249] Lustre: lustre-MDT0000: Denying connection for new client c6ccb568-5d68-4ea3-9e65-665cc299120d (at 192.168.202.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:19 [ 461.943863] Lustre: lustre-MDT0000: Denying connection for new client c6ccb568-5d68-4ea3-9e65-665cc299120d (at 192.168.202.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 461.952924] Lustre: Skipped 1 previous similar message [ 482.444582] Lustre: lustre-MDT0000: Denying connection for new client c6ccb568-5d68-4ea3-9e65-665cc299120d (at 192.168.202.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 482.458921] Lustre: Skipped 3 previous similar messages [ 518.270134] Lustre: lustre-MDT0000: Denying connection for new client c6ccb568-5d68-4ea3-9e65-665cc299120d (at 192.168.202.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:12 [ 518.289539] Lustre: Skipped 6 previous similar messages [ 531.000311] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 531.013200] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 531.038601] Lustre: lustre-MDT0000: Recovery over after 1:42, of 2 clients 1 recovered and 1 was evicted. [ 531.040305] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 531.059819] Lustre: Skipped 2 previous similar messages [ 531.093361] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:97 [ 531.103646] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:65 [ 541.850645] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 11:21:38 (1776180098) [ 547.817512] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 549.575802] Lustre: Failing over lustre-MDT0000 [ 549.775023] Lustre: server umount lustre-MDT0000 complete [ 549.872517] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.883901] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.892873] LustreError: Skipped 28 previous similar messages [ 562.144527] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180112/real 1776180112] req@000000001dc2bcbc x1862459339162368/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180119 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 562.167257] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.591347] LDISKFS-fs (dm-0): recovery complete [ 572.595866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.553407] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba134f5eb to 0x180b24bba134f9e9 [ 579.589680] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 579.783454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.719947] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 585.186146] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 585.519486] Lustre: lustre-MDT0000: Denying connection for new client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:41 [ 585.530137] Lustre: Skipped 2 previous similar messages [ 687.000352] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 687.010795] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 687.038031] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 687.038205] Lustre: lustre-MDT0000: Recovery over after 1:42, of 2 clients 1 recovered and 1 was evicted. [ 687.041391] Lustre: Skipped 3 previous similar messages [ 687.107621] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:129 [ 687.118147] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:97 [ 695.747887] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 11:24:12 (1776180252) [ 703.891657] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 706.004142] Lustre: Failing over lustre-MDT0000 [ 706.157702] Lustre: server umount lustre-MDT0000 complete [ 707.553797] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 707.576081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.608402] Lustre: Skipped 3 previous similar messages [ 707.614785] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.627942] LustreError: Skipped 34 previous similar messages [ 728.867460] LDISKFS-fs (dm-0): recovery complete [ 728.873495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.959833] LustreError: 11-0: MGC192.168.202.122@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 728.964107] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 728.985421] LustreError: 36:0:(import.c:702:ptlrpc_connect_import_locked()) already connecting [ 728.990470] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba134f9e9 to 0x180b24bba134fe6c [ 729.113060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 729.133463] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 729.207245] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 732.924482] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 734.258751] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 734.288879] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:129 [ 734.289690] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:161 [ 741.777700] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 743.987786] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 752.841352] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 11:25:08 (1776180308) [ 761.244886] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 763.338653] Lustre: Failing over lustre-MDT0000 [ 763.564522] Lustre: server umount lustre-MDT0000 complete [ 764.901397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.912053] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 764.926560] Lustre: Skipped 5 previous similar messages [ 772.067446] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180322/real 1776180322] req@000000005c9d8ac8 x1862459339211136/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180329 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 772.088717] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 787.878547] LDISKFS-fs (dm-0): recovery complete [ 787.881994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.487831] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba134fe6c to 0x180b24bba1350374 [ 789.519269] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 789.536721] Lustre: Skipped 5 previous similar messages [ 789.941393] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 790.046219] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 795.250437] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 795.289813] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:193 [ 795.290314] Lustre: lustre-OST0001: deleting orphan objects from 0x0:131 to 0x0:161 [ 795.852589] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 808.635639] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 809.892535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 819.403499] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 11:26:15 (1776180375) [ 827.744609] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 829.409046] Lustre: Failing over lustre-MDT0000 [ 829.795366] Lustre: server umount lustre-MDT0000 complete [ 830.945414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.960711] Lustre: Skipped 2 previous similar messages [ 838.112199] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180388/real 1776180388] req@000000003f9e9c4e x1862459339230208/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180395 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 838.143624] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 853.277723] LDISKFS-fs (dm-0): recovery complete [ 853.282713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.528092] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba1350374 to 0x180b24bba135089f [ 855.922943] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 856.881277] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 860.785980] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 861.367708] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 861.457975] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 861.459668] Lustre: lustre-OST0001: deleting orphan objects from 0x0:131 to 0x0:193 [ 870.404900] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 872.008820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 880.348689] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 11:27:16 (1776180436) [ 888.777384] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 890.780187] Lustre: Failing over lustre-MDT0000 [ 891.059183] Lustre: server umount lustre-MDT0000 complete [ 898.960168] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180449/real 1776180449] req@00000000c04520b8 x1862459339248064/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180456 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 898.984772] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 916.059218] LDISKFS-fs (dm-0): recovery complete [ 916.062794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.451128] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba135089f to 0x180b24bba1350dbc [ 916.594878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.602558] Lustre: Skipped 2 previous similar messages [ 916.644810] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 918.458960] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 920.749373] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 922.104093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 922.116528] Lustre: Skipped 10 previous similar messages [ 922.304991] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 922.355465] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 922.365213] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:225 [ 929.444106] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 930.764835] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 938.498698] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 11:28:14 (1776180494) [ 946.642963] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 952.805707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.826086] Lustre: Skipped 8 previous similar messages [ 960.480275] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180510/real 1776180510] req@000000009b4c65b3 x1862459339266368/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180517 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 960.513260] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 965.600890] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 965.616207] LustreError: Skipped 162 previous similar messages [ 973.586243] LDISKFS-fs (dm-0): recovery complete [ 973.592607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.880166] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba1350dbc to 0x180b24bba13512f5 [ 977.122254] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 982.381452] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 982.739098] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:289 [ 982.739392] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:257 [ 991.655104] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 993.159921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1000.159616] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 11:29:16 (1776180556) [ 1000.998671] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1001.001318] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1001.003779] LustreError: 16096:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000b10ed38e x1862459323392384/t38654705666(0) o35->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:29/0 lens 392/456 e 0 to 0 dl 1776180564 ref 1 fl Interpret:/0/0 rc 0/0 job:'openfile.0' [ 1008.106954] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1008.290258] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 1008.304193] Lustre: 16096:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000050d5a32b x1862459323392384/t38654705666(0) o35->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:36/0 lens 392/456 e 0 to 0 dl 1776180571 ref 1 fl Interpret:/2/0 rc 0/0 job:'openfile.0' [ 1009.566892] Lustre: Failing over lustre-MDT0000 [ 1009.568557] Lustre: Skipped 1 previous similar message [ 1009.697989] Lustre: server umount lustre-MDT0000 complete [ 1009.700237] Lustre: Skipped 1 previous similar message [ 1020.240168] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180570/real 1776180570] req@000000008bc3d441 x1862459339285184/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180577 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1030.850462] LDISKFS-fs (dm-0): recovery complete [ 1030.855314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.919210] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (not set up) [ 1037.211375] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1037.946394] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1037.953610] Lustre: Skipped 1 previous similar message [ 1042.351416] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1042.454455] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1042.463386] Lustre: Skipped 1 previous similar message [ 1042.514142] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:321 [ 1042.529630] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:289 [ 1051.586904] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1053.074175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1060.674856] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 11:30:17 (1776180617) [ 1067.963083] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1073.121060] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1080.289523] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180630/real 1776180630] req@0000000055435255 x1862459339302784/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180637 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 1080.318292] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1080.327673] LustreError: Skipped 1 previous similar message [ 1093.511773] LDISKFS-fs (dm-0): recovery complete [ 1093.527864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.704593] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba13517cc to 0x180b24bba1351c8e [ 1097.718353] Lustre: Skipped 1 previous similar message [ 1098.007886] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1102.802457] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1103.451672] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:321 [ 1103.452359] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:353 [ 1112.209653] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1113.924872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1121.999584] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 11:31:18 (1776180678) [ 1129.801598] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1130.682709] Lustre: *** cfs_fail_loc=114, val=0*** [ 1134.049695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.062573] Lustre: Skipped 9 previous similar messages [ 1155.918817] LDISKFS-fs (dm-0): recovery complete [ 1155.921884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1158.336631] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1162.377593] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1163.846941] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:353 [ 1163.849912] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:385 [ 1170.473492] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1171.944512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1178.700471] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 11:32:15 (1776180735) [ 1185.242105] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1186.106091] Lustre: *** cfs_fail_loc=128, val=0*** [ 1188.451730] Lustre: Failing over lustre-MDT0000 [ 1188.453449] Lustre: Skipped 2 previous similar messages [ 1188.575603] Lustre: server umount lustre-MDT0000 complete [ 1188.577935] Lustre: Skipped 2 previous similar messages [ 1189.350400] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1189.363093] LustreError: Skipped 1 previous similar message [ 1196.000540] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180746/real 1776180746] req@00000000c76a0049 x1862459339336128/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180753 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 1196.031657] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1210.396644] LDISKFS-fs (dm-0): recovery complete [ 1210.400170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.412582] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 1213.422131] Lustre: Skipped 23 previous similar messages [ 1213.593323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.596405] Lustre: Skipped 4 previous similar messages [ 1214.069899] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1214.076950] Lustre: Skipped 2 previous similar messages [ 1216.436235] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1219.091568] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1219.099860] Lustre: Skipped 2 previous similar messages [ 1219.125856] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:417 [ 1219.128805] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:385 [ 1224.690763] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1225.967570] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1232.610145] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 11:33:09 (1776180789) [ 1239.853631] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1251.747672] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1251.756121] LustreError: Skipped 2 previous similar messages [ 1266.702961] LDISKFS-fs (dm-0): recovery complete [ 1266.707467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.196432] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba1352619 to 0x180b24bba1352c4e [ 1268.204894] Lustre: Skipped 2 previous similar messages [ 1268.451857] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1268.466971] Lustre: Skipped 1 previous similar message [ 1272.768677] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1274.265919] Lustre: lustre-OST0001: deleting orphan objects from 0x0:391 to 0x0:417 [ 1274.270065] Lustre: lustre-OST0000: deleting orphan objects from 0x0:423 to 0x0:449 [ 1285.245793] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1287.624674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1296.766414] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 11:34:12 (1776180852) [ 1303.978962] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1329.247578] LDISKFS-fs (dm-0): recovery complete [ 1329.251547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.536196] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1340.124729] Lustre: lustre-OST0000: deleting orphan objects from 0x0:455 to 0x0:481 [ 1340.124951] Lustre: lustre-OST0001: deleting orphan objects from 0x0:423 to 0x0:449 [ 1351.526813] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1352.948749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1361.932314] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 11:35:18 (1776180918) [ 1369.854615] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1376.375594] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 1388.004945] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180938/real 1776180938] req@000000009cb28cfa x1862459339393088/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180945 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1388.041895] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1398.512800] LDISKFS-fs (dm-0): recovery complete [ 1398.516254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.775780] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1405.798308] Lustre: Skipped 1 previous similar message [ 1410.422877] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1414.328731] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:577 [ 1414.329186] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:609 [ 1420.130413] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1421.604800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1442.494560] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 11:36:38 (1776180998) [ 1449.430070] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1451.375944] Lustre: Failing over lustre-MDT0000 [ 1451.377462] Lustre: Skipped 3 previous similar messages [ 1451.511386] Lustre: server umount lustre-MDT0000 complete [ 1451.514144] Lustre: Skipped 3 previous similar messages [ 1452.007363] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.018394] Lustre: Skipped 21 previous similar messages [ 1473.619030] LDISKFS-fs (dm-0): recovery complete [ 1473.623956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.809827] LustreError: 11-0: MGC192.168.202.122@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 1473.821126] LustreError: Skipped 1 previous similar message [ 1473.824419] LustreError: 45911:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.202.122@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000014d8e38) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1477.753398] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1477.758535] Lustre: Skipped 3 previous similar messages [ 1479.078414] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1479.266233] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1479.275646] Lustre: Skipped 3 previous similar messages [ 1479.337955] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:641 [ 1479.347689] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:609 [ 1487.728463] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1489.135789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1498.973728] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 11:37:35 (1776181055) [ 1505.665950] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1508.512666] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.540975] LustreError: Skipped 303 previous similar messages [ 1517.536476] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1517.540349] LustreError: Skipped 3 previous similar messages [ 1529.041725] LDISKFS-fs (dm-0): recovery complete [ 1529.044821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.922019] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba135c4c6 to 0x180b24bba135c9ff [ 1533.937757] Lustre: Skipped 3 previous similar messages [ 1538.615513] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1539.724922] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:641 [ 1539.725719] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:673 [ 1547.700396] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1549.173783] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1557.426400] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 11:38:33 (1776181113) [ 1566.080352] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1593.916990] LDISKFS-fs (dm-0): recovery complete [ 1593.921062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.618476] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1615.991266] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:673 [ 1615.992642] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:705 [ 1623.961552] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1625.562212] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1633.302806] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 11:39:49 (1776181189) [ 1639.629143] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1653.728605] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181203/real 1776181203] req@00000000eb42e2c9 x1862459339501888/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181210 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 1653.743464] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1663.709214] LDISKFS-fs (dm-0): recovery complete [ 1663.713067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.473807] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1670.485472] Lustre: Skipped 3 previous similar messages [ 1674.688446] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1675.979939] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:705 [ 1675.984814] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:737 [ 1683.508656] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1684.840407] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1691.944606] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 11:40:48 (1776181248) [ 1698.661107] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1719.739649] LDISKFS-fs (dm-0): recovery complete [ 1719.741674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.642080] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1730.543950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 1730.554896] Lustre: Skipped 40 previous similar messages [ 1730.639159] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:737 [ 1730.643521] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:769 [ 1736.251820] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1737.575406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1744.714529] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 11:41:41 (1776181301) [ 1750.755957] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1756.129262] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1756.132402] LustreError: Skipped 1 previous similar message [ 1772.805807] LDISKFS-fs (dm-0): recovery complete [ 1772.808842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.855522] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (not set up) [ 1781.051917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1781.054929] Lustre: Skipped 8 previous similar messages [ 1786.482878] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:769 [ 1786.489455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:801 [ 1786.906459] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1798.128972] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1799.883284] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1808.083854] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 11:42:44 (1776181364) [ 1815.028584] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1839.979461] LDISKFS-fs (dm-0): recovery complete [ 1839.982925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1846.721823] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1847.369240] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:833 [ 1847.373703] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:801 [ 1857.402877] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1859.507351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1870.500652] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 11:43:46 (1776181426) [ 1878.202422] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1902.849464] LDISKFS-fs (dm-0): recovery complete [ 1902.852537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.161605] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1913.395512] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:833 [ 1913.398781] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:865 [ 1919.984736] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1921.471696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1928.579477] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 11:44:45 (1776181485) [ 1937.342820] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1965.099143] LDISKFS-fs (dm-0): recovery complete [ 1965.104829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.411532] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1974.356259] Lustre: lustre-OST0000: deleting orphan objects from 0x0:867 to 0x0:897 [ 1974.356570] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:865 [ 1983.305417] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1984.937414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1993.067399] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 11:45:49 (1776181549) [ 1999.259234] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2000.684235] Lustre: Failing over lustre-MDT0000 [ 2000.686329] Lustre: Skipped 8 previous similar messages [ 2000.784375] Lustre: server umount lustre-MDT0000 complete [ 2000.786279] Lustre: Skipped 8 previous similar messages [ 2004.960957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.967970] Lustre: Skipped 33 previous similar messages [ 2023.350341] LDISKFS-fs (dm-0): recovery complete [ 2023.356854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.652936] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2031.659229] Lustre: Skipped 8 previous similar messages [ 2034.165741] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2035.246792] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2035.251714] Lustre: Skipped 8 previous similar messages [ 2035.272825] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:897 [ 2035.273187] Lustre: lustre-OST0000: deleting orphan objects from 0x0:899 to 0x0:929 [ 2042.476020] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2044.072584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2053.310971] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 11:46:49 (1776181609) [ 2061.470426] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2072.993377] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2073.024088] LustreError: Skipped 8 previous similar messages [ 2084.995355] LDISKFS-fs (dm-0): recovery complete [ 2084.998044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.466084] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba135f55d to 0x180b24bba135fa88 [ 2090.485232] Lustre: Skipped 8 previous similar messages [ 2095.361270] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2096.259912] Lustre: lustre-OST0001: deleting orphan objects from 0x0:899 to 0x0:929 [ 2096.261167] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:961 [ 2103.260480] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2104.448224] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2112.348209] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 11:47:48 (1776181668) [ 2119.187446] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2121.716259] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.727721] LustreError: Skipped 397 previous similar messages [ 2144.414290] LDISKFS-fs (dm-0): recovery complete [ 2144.422929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2154.965746] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2156.639093] Lustre: lustre-OST0001: deleting orphan objects from 0x0:931 to 0x0:961 [ 2156.653797] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:993 [ 2162.832458] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2164.155697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2171.288079] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 11:48:47 (1776181727) [ 2179.728704] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2189.264878] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181739/real 1776181739] req@0000000082b63fc4 x1862459339659136/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181746 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 2189.290414] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2204.438967] LDISKFS-fs (dm-0): recovery complete [ 2204.442898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2206.974852] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2206.993859] Lustre: Skipped 8 previous similar messages [ 2210.757935] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2212.399947] Lustre: lustre-OST0001: deleting orphan objects from 0x0:963 to 0x0:993 [ 2212.408380] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:1025 [ 2217.382631] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2218.835139] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2227.851666] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 11:49:43 (1776181783) [ 2237.551974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2264.875108] LDISKFS-fs (dm-0): recovery complete [ 2264.881392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2267.268883] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (not set up) [ 2272.216069] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2272.922425] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2281.604572] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2283.109496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2291.441576] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 11:50:48 (1776181848) [ 2298.834788] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2323.062775] LDISKFS-fs (dm-0): recovery complete [ 2323.064725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.693227] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (not set up) [ 2332.578777] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2334.187703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2334.204242] Lustre: Skipped 49 previous similar messages [ 2334.301399] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1025 [ 2334.307109] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1060 to 0x0:1089 [ 2340.550832] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2341.497883] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2346.866333] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:51:43 (1776181903) [ 2354.227468] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2380.639969] LDISKFS-fs (dm-0): recovery complete [ 2380.656609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.644374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.650150] Lustre: Skipped 9 previous similar messages [ 2389.373796] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2390.124456] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1091 to 0x0:1121 [ 2390.126123] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1057 [ 2398.225148] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2399.550186] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2407.077531] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 11:52:43 (1776181963) [ 2410.066387] Lustre: 77544:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 00c2ea12-0d37-44cd-bf29-21c9c37834bb at adminstrative request [ 2434.606036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.129916] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2444.834388] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1089 [ 2444.834514] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 2452.235383] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2453.535973] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2459.014501] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2466.587034] Lustre: DEBUG MARKER: before 2844, after 2844 [ 2472.037296] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 11:53:48 (1776182028) [ 2473.051315] Lustre: 79579:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 00c2ea12-0d37-44cd-bf29-21c9c37834bb at adminstrative request [ 2480.944191] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:53:57 (1776182037) [ 2487.117081] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2510.703058] LDISKFS-fs (dm-0): recovery complete [ 2510.705811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2519.905991] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2521.216308] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1091 to 0x0:1121 [ 2521.217278] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1156 to 0x0:1185 [ 2527.788170] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2529.308442] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2537.624954] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:54:54 (1776182094) [ 2544.322739] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2568.576333] LDISKFS-fs (dm-0): recovery complete [ 2568.579299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2582.132405] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2582.167379] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 2582.167759] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2593.473428] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2594.832577] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2601.832877] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:55:58 (1776182158) [ 2608.451412] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2609.997789] Lustre: Failing over lustre-MDT0000 [ 2609.999459] Lustre: Skipped 9 previous similar messages [ 2610.150380] Lustre: server umount lustre-MDT0000 complete [ 2610.152971] Lustre: Skipped 9 previous similar messages [ 2612.706698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2612.729157] Lustre: Skipped 42 previous similar messages [ 2630.736501] LDISKFS-fs (dm-0): recovery complete [ 2630.746708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2636.920913] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2636.925270] Lustre: Skipped 9 previous similar messages [ 2640.266255] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2641.972245] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2641.982452] Lustre: Skipped 9 previous similar messages [ 2642.032450] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2642.032975] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 2647.525611] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2648.891779] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2657.374300] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 11:56:53 (1776182213) [ 2664.021090] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2667.490229] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2667.494602] LustreError: Skipped 4 previous similar messages [ 2674.658066] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2674.666448] LustreError: Skipped 9 previous similar messages [ 2687.856148] LDISKFS-fs (dm-0): recovery complete [ 2687.859142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2691.121222] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba1362b03 to 0x180b24bba1363051 [ 2691.133764] Lustre: Skipped 9 previous similar messages [ 2691.191603] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (not set up) [ 2696.215522] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2696.770246] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2696.770514] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2704.744475] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2705.952604] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2713.253844] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:57:49 (1776182269) [ 2719.950579] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2722.275653] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2722.306309] LustreError: Skipped 381 previous similar messages [ 2744.208808] LDISKFS-fs (dm-0): recovery complete [ 2744.211795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.701314] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2751.519829] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2751.531127] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1313 [ 2758.998630] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2760.929495] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2768.414578] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:58:45 (1776182325) [ 2775.105741] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2789.344260] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182339/real 1776182339] req@00000000bd5a5914 x1862459339833344/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182346 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 2789.373219] Lustre: 3354:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 2799.303952] LDISKFS-fs (dm-0): recovery complete [ 2799.307756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.983452] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2806.996349] Lustre: Skipped 9 previous similar messages [ 2811.123730] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2812.634140] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2812.634700] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2819.940646] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2821.366974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2828.720439] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:59:45 (1776182385) [ 2834.555107] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2860.095480] LDISKFS-fs (dm-0): recovery complete [ 2860.098251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.356075] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2868.359936] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 2868.618902] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2877.899819] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2879.409719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2885.664058] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:00:42 (1776182442) [ 2892.254487] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2915.857975] LDISKFS-fs (dm-0): recovery complete [ 2915.860495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.274026] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2929.302477] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2929.305921] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 2936.874505] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2938.583383] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2947.363935] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:01:43 (1776182503) [ 2953.554151] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2983.646724] LDISKFS-fs (dm-0): recovery complete [ 2983.655021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2984.427438] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 2984.429991] Lustre: Skipped 53 previous similar messages [ 2984.689150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.697123] Lustre: Skipped 9 previous similar messages [ 2989.803674] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2990.177877] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2990.179644] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 2999.672874] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3000.963458] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3009.636404] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:02:46 (1776182566) [ 3017.302363] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3043.711732] LDISKFS-fs (dm-0): recovery complete [ 3043.715757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3049.658343] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3051.041700] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 3051.041877] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 3058.515417] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3059.923342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3067.017660] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 12:03:43 (1776182623) [ 3074.324775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3098.281037] LDISKFS-fs (dm-0): recovery complete [ 3098.288930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.212509] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3110.994092] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1475 to 0x0:1505 [ 3110.994654] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 3118.824752] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3120.410714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3128.887654] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 12:04:45 (1776182685) [ 3130.242809] Lustre: 102291:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 00c2ea12-0d37-44cd-bf29-21c9c37834bb at adminstrative request [ 3140.484782] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 12:04:56 (1776182696) [ 3147.482587] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3161.416174] LDISKFS-fs (dm-0): recovery complete [ 3161.420459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.300656] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (not set up) [ 3165.471460] LustreError: 103574:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3165.484656] LustreError: 103574:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3165.500248] Lustre: 103607:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3165.510484] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3165.518453] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3165.540279] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3165.618631] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1447 to 0x0:1473 [ 3165.624018] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1512 to 0x0:1537 [ 3170.564124] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3170.788828] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3192.802925] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 12:05:48 (1776182748) [ 3200.190364] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3214.079045] LDISKFS-fs (dm-0): recovery complete [ 3214.082806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.201892] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3220.223938] LustreError: 105490:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3220.229421] LustreError: 105490:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3220.235186] Lustre: 105525:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3220.238068] Lustre: 105525:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3220.243615] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3220.250155] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3220.267540] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3220.325440] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1548 to 0x0:1569 [ 3220.329487] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1484 to 0x0:1505 [ 3225.604143] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3225.628794] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3240.502122] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3240.503872] Lustre: Skipped 1 previous similar message [ 3248.842275] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 12:06:45 (1776182805) [ 3258.259888] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3260.041703] Lustre: Failing over lustre-MDT0000 [ 3260.045204] Lustre: Skipped 10 previous similar messages [ 3260.264530] Lustre: server umount lustre-MDT0000 complete [ 3260.269166] Lustre: Skipped 10 previous similar messages [ 3261.420794] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3261.452253] Lustre: Skipped 42 previous similar messages [ 3274.470993] LDISKFS-fs (dm-0): recovery complete [ 3274.473542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.106859] LustreError: 107412:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3275.115708] LustreError: 107412:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3275.135148] Lustre: 107445:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3275.142469] Lustre: 107445:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3275.156934] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3275.164580] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3275.187295] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3275.247518] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1512 to 0x0:1537 [ 3275.256569] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1601 [ 3280.370228] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3281.380808] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3302.916283] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 12:07:39 (1776182859) [ 3304.119719] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3304.124470] LustreError: 6255:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000090d247eb x1862459323936192/t201863462916(0) o36->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:101/0 lens 504/456 e 0 to 0 dl 1776182901 ref 1 fl Interpret:/0/0 rc 0/0 job:'rm.0' [ 3318.181787] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3318.185700] LustreError: Skipped 10 previous similar messages [ 3318.304749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3323.000272] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.016833] LustreError: Skipped 358 previous similar messages [ 3324.397412] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba1367483 to 0x180b24bba13678d5 [ 3324.424892] Lustre: Skipped 10 previous similar messages [ 3324.805831] LustreError: 108790:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3324.810476] LustreError: 108790:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3324.820134] Lustre: 108824:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3324.831447] Lustre: 108824:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3324.837809] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3324.844470] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3324.879624] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3325.007849] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1633 [ 3325.014724] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1539 to 0x0:1569 [ 3329.833324] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3330.038974] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3350.234801] Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ============ 12:08:26 (1776182906) [ 3358.555174] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3360.748336] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3360.762188] LustreError: Skipped 5 previous similar messages [ 3384.491526] LDISKFS-fs (dm-0): recovery complete [ 3384.498010] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.011098] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3392.014346] Lustre: Skipped 8 previous similar messages [ 3395.835378] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3396.176850] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3396.185768] Lustre: Skipped 8 previous similar messages [ 3396.250668] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1571 to 0x0:1601 [ 3396.252531] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1665 [ 3405.664412] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 12:09:21 (1776182961) [ 3413.935536] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3416.561324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.568213] Lustre: Skipped 3 previous similar messages [ 3428.768133] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182978/real 1776182978] req@0000000032923cf1 x1862459340032640/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182985 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 3428.799107] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 3428.881806] LDISKFS-fs (dm-0): recovery complete [ 3428.884719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.267770] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3435.268444] LustreError: 112558:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3435.272662] Lustre: Skipped 18 previous similar messages [ 3435.288609] LustreError: 112558:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3435.295626] Lustre: 112592:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3435.304814] Lustre: 112592:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3435.319188] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3435.327942] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3435.353038] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3435.393253] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1697 [ 3435.404773] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1571 to 0x0:1633 [ 3440.408183] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3440.618393] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3462.844741] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:10:19 (1776183019) [ 3491.445432] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3519.403079] LDISKFS-fs (dm-0): recovery complete [ 3519.404783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.463218] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3527.329617] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2034 to 0x0:2049 [ 3527.331257] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2098 to 0x0:2113 [ 3536.073599] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3537.818076] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3555.301137] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:11:51 (1776183111) [ 3575.179878] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3606.833151] LDISKFS-fs (dm-0): recovery complete [ 3606.838816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3608.056323] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 3608.065149] Lustre: Skipped 49 previous similar messages [ 3608.259824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3608.267918] Lustre: Skipped 9 previous similar messages [ 3612.398153] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3616.668149] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2514 to 0x0:2529 [ 3616.689785] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2465 [ 3622.114721] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3623.707898] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3643.608141] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 12:13:19 (1776183199) [ 3645.398509] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 3647.624519] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 12:13:23 (1776183203) [ 3650.140824] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3651.238654] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3651.252529] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3651.270491] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2497 [ 3658.141929] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 12:13:34 (1776183214) [ 3677.043804] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3713.011679] LDISKFS-fs (dm-2): recovery complete [ 3713.014757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3715.331372] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3715.361397] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:2977 [ 3718.165807] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3772.584974] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 12:15:29 (1776183329) [ 3779.196758] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3802.696221] LDISKFS-fs (dm-0): recovery complete [ 3802.702820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.073666] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3815.501336] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3815.502140] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3009 [ 3822.561193] LustreError: 122375:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3822.571249] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3823.587033] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2913 [ 3823.847943] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3825.397988] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3844.620298] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 12:16:41 (1776183401) [ 3849.187138] LustreError: 9943:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3854.304121] LustreError: 9943:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3854.312173] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 3854.371278] LustreError: 8226:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3856.073991] LustreError: 9943:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3861.472357] LustreError: 9943:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3861.480911] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 3863.490784] LustreError: 16195:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3868.644500] LustreError: 16195:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3868.652512] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 3870.437821] LustreError: 9943:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3875.808144] LustreError: 9943:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3877.667601] LustreError: 16195:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3882.976222] LustreError: 16195:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3882.996906] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 3883.012747] Lustre: Skipped 1 previous similar message [ 3892.060505] LustreError: 6256:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3892.065691] LustreError: 6256:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message [ 3897.313747] LustreError: 6256:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3897.323675] LustreError: 6256:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message [ 3897.353361] LustreError: 10023:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3897.357907] LustreError: 10023:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 3904.993094] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 3905.007533] Lustre: Skipped 3 previous similar messages [ 3913.801557] LustreError: 16195:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3913.808301] LustreError: 16195:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages [ 3919.328468] LustreError: 16195:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3919.339312] LustreError: 16195:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages [ 3928.160837] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 12:18:04 (1776183484) [ 3930.364199] LustreError: 6254:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3937.913337] Lustre: lustre-MDT0000: Export 0000000016be1f95 already connecting from 192.168.202.22@tcp [ 3939.978608] Lustre: lustre-MDT0000: Export 0000000016be1f95 already connecting from 192.168.202.22@tcp [ 3941.824927] Lustre: lustre-MDT0000: Export 0000000016be1f95 already connecting from 192.168.202.22@tcp [ 3945.647870] Lustre: lustre-MDT0000: Export 0000000016be1f95 already connecting from 192.168.202.22@tcp [ 3945.660090] Lustre: Skipped 2 previous similar messages [ 3951.553612] Lustre: lustre-MDT0000: Export 0000000016be1f95 already connecting from 192.168.202.22@tcp [ 3951.558311] Lustre: Skipped 3 previous similar messages [ 3956.304289] LustreError: 6254:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 3956.312792] Lustre: 6254:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/6s); client may timeout req@000000006ed6c7a3 x1862459324968960/t0(0) o38->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:0/0 lens 520/416 e 0 to 0 dl 1776183507 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0' [ 3958.395921] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 3958.406121] Lustre: Skipped 3 previous similar messages [ 3960.954870] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 12:18:37 (1776183517) [ 3967.781271] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3971.199903] Lustre: Failing over lustre-MDT0000 [ 3971.210529] Lustre: Skipped 7 previous similar messages [ 3971.467980] Lustre: server umount lustre-MDT0000 complete [ 3971.471222] Lustre: Skipped 7 previous similar messages [ 3973.769948] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.784393] LustreError: Skipped 217 previous similar messages [ 3974.113215] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3974.117535] LustreError: Skipped 2 previous similar messages [ 3974.121514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3974.129535] Lustre: Skipped 29 previous similar messages [ 3982.881465] LDISKFS-fs (dm-0): recovery complete [ 3982.885231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.002251] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3983.008552] LustreError: Skipped 5 previous similar messages [ 3983.017531] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba139a1d3 to 0x180b24bba139ab6c [ 3983.023319] LustreError: 127049:0:(import.c:702:ptlrpc_connect_import_locked()) already connecting [ 3983.023802] Lustre: Skipped 5 previous similar messages [ 3983.185971] Lustre: *** cfs_fail_loc=712, val=0*** [ 3983.195138] LustreError: 44539:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery req@0000000031dfc1da x1862459340376640/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' [ 3983.227952] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3983.322176] LustreError: 127049:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3983.329414] LustreError: 127049:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3983.335819] Lustre: 127083:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3983.341442] Lustre: 127083:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3983.347450] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3983.357608] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0] [ 3983.382334] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3983.447107] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2945 [ 3983.452480] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3041 [ 3987.634787] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3988.460630] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4025.231333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.641051] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4035.661291] Lustre: Skipped 4 previous similar messages [ 4038.156267] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4039.175959] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4039.188732] Lustre: Skipped 4 previous similar messages [ 4039.232156] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3073 [ 4039.235313] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2977 [ 4048.007616] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4049.524132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4057.626244] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:20:14 (1776183614) [ 4057.879044] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 4065.323333] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:20:21 (1776183621) [ 4066.233406] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4066.236879] LustreError: 18174:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000064a2cc9c x1862459324995328/t0(0) o700->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:74/0 lens 264/248 e 0 to 0 dl 1776183629 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0' [ 4087.264164] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183637/real 1776183637] req@00000000eff72a37 x1862459340406464/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183644 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 4087.293810] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 4093.646234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.910755] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4104.919709] Lustre: Skipped 10 previous similar messages [ 4109.066080] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4110.365515] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2979 to 0x0:3009 [ 4110.366512] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3075 to 0x0:3105 [ 4117.942155] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4119.645979] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4130.112158] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:21:25 (1776183685) [ 4151.989897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4153.529645] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97 [ 4153.539597] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137 [ 4156.501592] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4165.196300] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4166.914464] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4241.510954] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:23:17 (1776183797) [ 4249.617301] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4272.242842] LDISKFS-fs (dm-0): recovery complete [ 4272.245075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.248882] Lustre: MGC192.168.202.122@tcp: Connection restored to 192.168.202.122@tcp (at 0@lo) [ 4278.253956] Lustre: Skipped 31 previous similar messages [ 4278.393447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4278.396810] Lustre: Skipped 6 previous similar messages [ 4281.930304] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4284.281522] Lustre: *** cfs_fail_loc=216, val=0*** [ 4284.281718] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3030 to 0x0:3073 [ 4284.283024] LustreError: 133510:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 4285.345134] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169 [ 4353.081644] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:25:09 (1776183909) [ 4354.654404] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4354.658831] Lustre: Skipped 2 previous similar messages [ 4354.665918] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201 [ 4355.511615] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3233 [ 4366.346364] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:25:22 (1776183922) [ 4387.356691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.881436] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4401.131814] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4401.142344] LustreError: 135194:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000bbce7b15 x1862459325051008/t0(0) o101->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:438/0 lens 328/344 e 0 to 0 dl 1776183993 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0' [ 4437.629320] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 4437.717426] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3235 to 0x0:3265 [ 4437.718806] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3084 to 0x0:3105 [ 4443.291611] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4444.737548] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4452.772586] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:26:49 (1776184009) [ 4454.759937] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4461.593965] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4485.107060] LDISKFS-fs (dm-0): recovery complete [ 4485.118271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.571938] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4497.023260] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3297 [ 4497.026242] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3084 to 0x0:3137 [ 4504.331874] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4505.892371] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4513.841564] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:27:50 (1776184070) [ 4514.904468] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4522.307485] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4547.201405] LDISKFS-fs (dm-0): recovery complete [ 4547.205958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.295949] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (not set up) [ 4555.518969] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4556.864737] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3329 [ 4556.869452] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3169 [ 4564.973700] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4566.748900] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4575.902537] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:28:52 (1776184132) [ 4577.064505] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4586.481970] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4588.524283] Lustre: Failing over lustre-MDT0000 [ 4588.525951] Lustre: Skipped 7 previous similar messages [ 4590.704352] Lustre: server umount lustre-MDT0000 complete [ 4590.710163] Lustre: Skipped 7 previous similar messages [ 4591.259355] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4591.284663] LustreError: Skipped 263 previous similar messages [ 4592.611734] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4592.616131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4592.621128] LustreError: Skipped 7 previous similar messages [ 4592.644623] Lustre: Skipped 34 previous similar messages [ 4599.776191] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4599.791939] LustreError: Skipped 6 previous similar messages [ 4613.044198] LDISKFS-fs (dm-0): recovery complete [ 4613.048551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.199592] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba139e303 to 0x180b24bba139e9b6 [ 4617.212741] Lustre: Skipped 6 previous similar messages [ 4621.902420] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4622.948948] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3361 [ 4622.950033] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3201 [ 4632.974667] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:29:49 (1776184189) [ 4635.228934] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4635.236154] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4635.239597] LustreError: 6258:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000ec992fa2 x1862459325081856/t261993005072(0) o35->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:643/0 lens 392/456 e 0 to 0 dl 1776184198 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4638.182407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4638.187690] Lustre: Skipped 2 previous similar messages [ 4659.888672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.532849] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4668.550900] Lustre: Skipped 7 previous similar messages [ 4672.504574] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4672.515177] Lustre: Skipped 7 previous similar messages [ 4672.527144] Lustre: 6258:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000d51b4533 x1862459325081856/t261993005072(0) o35->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:680/0 lens 392/456 e 0 to 0 dl 1776184235 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4672.563133] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3363 to 0x0:3393 [ 4672.563402] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3233 [ 4672.671586] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4681.351760] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4683.762819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4695.771260] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:30:51 (1776184251) [ 4697.684050] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4697.686153] LustreError: 6254:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000085dd0dcc x1862459325090816/t266287972368(0) o36->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:739/0 lens 504/448 e 0 to 0 dl 1776184294 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4708.580388] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4720.608451] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184270/real 1776184270] req@000000004be883ae x1862459340583744/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184277 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 4720.627450] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 4733.227274] LDISKFS-fs (dm-0): recovery complete [ 4733.229877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4737.310172] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4737.314578] Lustre: Skipped 7 previous similar messages [ 4742.399733] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4742.750958] Lustre: 6254:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000085dd15c7 x1862459325090816/t266287972368(0) o36->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:29/0 lens 504/448 e 0 to 0 dl 1776184339 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4742.787597] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425 [ 4742.791412] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3265 [ 4752.226131] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4753.990814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4763.518968] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:31:59 (1776184319) [ 4764.840479] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4764.848216] LustreError: 9943:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000063a97925 x1862459325099712/t270582939664(0) o36->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:52/0 lens 504/448 e 0 to 0 dl 1776184362 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4766.837375] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4774.004370] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 4774.014311] Lustre: Skipped 3 previous similar messages [ 4774.029865] Lustre: 6258:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000134e7540 x1862459325099840/t270582939665(0) o35->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:27/0 lens 392/456 e 0 to 0 dl 1776184337 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4774.048791] Lustre: 6258:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4774.578570] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4801.584775] LDISKFS-fs (dm-0): recovery complete [ 4801.587138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4808.657783] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4808.715950] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3297 [ 4808.722985] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3457 [ 4820.684862] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:32:56 (1776184376) [ 4821.943043] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4821.945114] Lustre: Skipped 1 previous similar message [ 4821.954017] LustreError: 9943:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000007a90a049 x1862459325107776/t274877906960(0) o36->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:109/0 lens 504/448 e 0 to 0 dl 1776184419 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4822.007615] LustreError: 9943:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 4824.109474] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4824.114323] Lustre: Skipped 1 previous similar message [ 4831.368649] Lustre: 6255:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000c83f2f9f x1862459325107776/t274877906960(0) o36->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:118/0 lens 504/448 e 0 to 0 dl 1776184428 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4832.980915] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4858.908814] LDISKFS-fs (dm-0): recovery complete [ 4858.913037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4869.300498] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3329 [ 4869.303449] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3489 [ 4869.525419] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4881.678233] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:33:57 (1776184437) [ 4882.896902] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4885.070350] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4885.080565] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4885.097284] LustreError: 6258:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000076b7deff x1862459325115968/t279172874256(0) o35->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:138/0 lens 392/456 e 0 to 0 dl 1776184448 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4892.345176] Lustre: 6258:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000009baa5860 x1862459325115968/t279172874256(0) o35->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:145/0 lens 392/456 e 0 to 0 dl 1776184455 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4894.556110] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4919.570333] LDISKFS-fs (dm-0): recovery complete [ 4919.572535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.882748] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 4923.891438] Lustre: Skipped 46 previous similar messages [ 4924.145089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4924.162102] Lustre: Skipped 8 previous similar messages [ 4928.663545] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4929.677431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3521 [ 4929.683701] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3361 [ 4939.646421] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:34:56 (1776184496) [ 4940.535589] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4940.542072] LustreError: 16195:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000d75f79ae x1862459325122944/t283467841549(0) o101->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:227/0 lens 664/600 e 0 to 0 dl 1776184537 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0' [ 4982.395812] Lustre: 6254:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000008a705cd3 x1862459325122944/t283467841549(0) o101->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:269/0 lens 664/3424 e 0 to 0 dl 1776184579 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0' [ 4989.046642] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:35:45 (1776184545) [ 4996.613971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5022.862039] LDISKFS-fs (dm-0): recovery complete [ 5022.868607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5030.856620] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5031.525243] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3393 [ 5031.526107] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3553 [ 5040.206569] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5041.938227] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5061.296773] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:36:57 (1776184617) [ 5070.209881] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5072.353520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5097.175947] LDISKFS-fs (dm-0): recovery complete [ 5097.179943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.641517] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5107.853641] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3555 to 0x0:3585 [ 5107.856387] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3425 [ 5118.990311] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5120.881923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5127.021573] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5139.794134] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:38:15 (1776184695) [ 5183.154117] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5191.289517] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5191.303733] LustreError: Skipped 319 previous similar messages [ 5211.914854] LDISKFS-fs (dm-0): recovery complete [ 5211.920460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5219.545121] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5220.228943] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4865 [ 5220.230851] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4676 to 0x0:4705 [ 5230.794885] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5232.641780] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5305.772858] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:41:01 (1776184861) [ 5316.235410] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5318.345628] Lustre: Failing over lustre-MDT0000 [ 5318.350659] Lustre: Skipped 8 previous similar messages [ 5318.616377] Lustre: server umount lustre-MDT0000 complete [ 5318.619977] Lustre: Skipped 8 previous similar messages [ 5322.219867] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.237380] Lustre: Skipped 34 previous similar messages [ 5329.376207] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184879/real 1776184879] req@000000001a373acb x1862459341073472/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184886 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 5329.406511] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 5329.429354] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5329.441491] LustreError: Skipped 8 previous similar messages [ 5344.848773] LDISKFS-fs (dm-0): recovery complete [ 5344.853377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.858523] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba13b7058 to 0x180b24bba13e21ed [ 5346.888152] Lustre: Skipped 8 previous similar messages [ 5347.290937] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5347.309251] Lustre: Skipped 6 previous similar messages [ 5347.959101] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5347.969787] Lustre: Skipped 7 previous similar messages [ 5352.597718] Lustre: lustre-MDT0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 5352.611158] Lustre: Skipped 7 previous similar messages [ 5352.662636] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4676 to 0x0:4737 [ 5352.666831] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4867 to 0x0:4897 [ 5353.624589] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5364.874285] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5366.421553] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5377.819653] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5379.721962] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5386.874545] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:42:23 (1776184943) [ 5394.583706] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5437.048182] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 5437.068680] Lustre: Skipped 3 previous similar messages [ 5440.583663] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5440.587513] Lustre: Skipped 1 previous similar message [ 5440.591545] LustreError: 6255:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000004edb9b4 x1862459326614272/t300647710728(0) o36->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:727/0 lens 66040/440 e 0 to 0 dl 1776185037 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0' [ 5482.119173] Lustre: 6255:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000dda90c37 x1862459326614272/t300647710728(0) o36->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:14/0 lens 66040/440 e 0 to 0 dl 1776185079 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0' [ 5497.397140] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5499.811531] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:44:15 (1776185055) [ 5516.269371] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5542.758388] LDISKFS-fs (dm-0): recovery complete [ 5542.765284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5544.934474] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 5544.941328] Lustre: Skipped 24 previous similar messages [ 5545.239987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5545.251499] Lustre: Skipped 4 previous similar messages [ 5549.827642] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5552.104984] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4839 to 0x0:4865 [ 5552.111972] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4998 to 0x0:5025 [ 5560.314927] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5562.224181] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5573.378898] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:45:29 (1776185129) [ 5596.412205] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5636.575417] LDISKFS-fs (dm-2): recovery complete [ 5636.578688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5639.975509] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5441 [ 5639.990575] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129 [ 5642.986756] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5660.645733] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5660.666688] LustreError: Skipped 7 previous similar messages [ 5677.224209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5678.697538] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161 [ 5678.729307] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5473 [ 5681.986724] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5691.796771] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5693.596542] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5736.654875] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:48:12 (1776185292) [ 5757.994073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.822232] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (not set up) [ 5768.840032] Lustre: Skipped 3 previous similar messages [ 5774.180900] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5774.377929] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5266 to 0x0:5281 [ 5774.379343] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5505 [ 5794.788663] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5794.799974] LustreError: Skipped 185 previous similar messages [ 5808.911652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.119752] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5818.948928] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5537 [ 5818.954434] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5266 to 0x0:5313 [ 5826.892841] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5828.264342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5836.612035] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:49:52 (1776185392) [ 5871.398569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5873.981896] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5569 [ 5873.990382] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:193 [ 5877.256246] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5886.714326] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5888.682964] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5897.818451] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:50:54 (1776185454) [ 5909.562697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5909.648339] Lustre: *** cfs_fail_loc=605, val=0*** [ 5909.650949] LustreError: 172408:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=0000000050f56f12 failed: rc = -95 [ 5909.659903] LustreError: 172408:0:(obd_config.c:774:class_setup()) setup MGS failed (-95) [ 5909.665078] LustreError: 172408:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95 [ 5909.672593] LustreError: 172408:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered [ 5909.678729] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5909.684796] LustreError: 172408:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 5909.696803] LustreError: 172408:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5918.960693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5924.951617] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5315 to 0x0:5345 [ 5924.956290] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5601 [ 5925.461974] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5934.622836] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:51:30 (1776185490) [ 5942.254605] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5945.656079] Lustre: Failing over lustre-MDT0000 [ 5945.663435] Lustre: Skipped 7 previous similar messages [ 5945.870737] Lustre: server umount lustre-MDT0000 complete [ 5945.872813] Lustre: Skipped 8 previous similar messages [ 5950.435049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.447881] Lustre: Skipped 23 previous similar messages [ 5958.112210] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185507/real 1776185507] req@00000000029e9ba4 x1862459341310080/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185514 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 5958.147661] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5958.157402] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5958.165973] LustreError: Skipped 4 previous similar messages [ 5969.087219] LDISKFS-fs (dm-0): recovery complete [ 5969.090793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5975.523845] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba13f6db7 to 0x180b24bba13f75eb [ 5975.534881] Lustre: Skipped 4 previous similar messages [ 5975.669401] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (not set up) [ 5975.821877] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5975.832869] Lustre: Skipped 7 previous similar messages [ 5977.131842] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5977.143881] Lustre: Skipped 7 previous similar messages [ 5977.155171] Lustre: *** cfs_fail_loc=707, val=0*** [ 5979.846720] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6018.709237] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 6019.130676] Lustre: lustre-MDT0000: Recovery over after 0:42, of 2 clients 2 recovered and 0 were evicted. [ 6019.137626] Lustre: Skipped 7 previous similar messages [ 6019.204054] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5359 to 0x0:5377 [ 6019.204627] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5614 to 0x0:5633 [ 6024.566638] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6026.121629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6036.591620] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:53:12 (1776185592) [ 6068.352345] LustreError: 6255:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 6074.400223] LustreError: 6255:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6094.649926] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:54:10 (1776185650) [ 6125.134183] LustreError: 43140:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms [ 6131.184181] LustreError: 43140:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake [ 6139.886771] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:54:56 (1776185696) [ 6171.580286] LustreError: 6256:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 6176.680129] LustreError: 6256:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6179.123846] LustreError: 6256:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 6189.144441] LustreError: 6256:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6208.525325] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 12:56:04 (1776185764) [ 6269.229739] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 12:57:05 (1776185825) [ 6303.816196] LustreError: 10023:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6304.248160] LustreError: 10023:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6311.904939] LustreError: 10021:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6311.910592] LustreError: 10021:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 28 previous similar messages [ 6312.328558] LustreError: 8230:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6312.346161] LustreError: 8230:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 28 previous similar messages [ 6327.960510] LustreError: 6256:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6327.969436] LustreError: 6256:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 63 previous similar messages [ 6328.400140] LustreError: 6256:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6328.413562] LustreError: 6256:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 62 previous similar messages [ 6336.527845] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 12:58:13 (1776185893) [ 6369.735250] Lustre: DEBUG MARKER: phase 2 [ 6380.769907] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 12:58:57 (1776185937) [ 6463.482368] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 13:00:20 (1776186020) [ 6464.667467] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6466.493611] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 13:00:22 (1776186022) [ 6470.617608] Lustre: DEBUG MARKER: Started rundbench load pid=138656 ... [ 6479.989043] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6482.783648] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6487.533805] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6487.538397] LustreError: Skipped 5 previous similar messages [ 6487.562048] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6487.591062] LustreError: Skipped 106 previous similar messages [ 6512.412808] LDISKFS-fs (dm-0): recovery complete [ 6512.416334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6512.633376] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 6512.643673] Lustre: Skipped 30 previous similar messages [ 6512.755178] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (not set up) [ 6512.918703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6512.923048] Lustre: Skipped 7 previous similar messages [ 6517.531303] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6518.377099] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5702 to 0x0:5729 [ 6518.382837] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5426 to 0x0:5441 [ 6527.325832] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6528.891272] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6541.162702] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6544.678190] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6547.209746] Lustre: Failing over lustre-MDT0001 [ 6547.219603] Lustre: Skipped 1 previous similar message [ 6547.238932] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.22@tcp (stopping) [ 6547.264402] LustreError: 9943:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export 00000000b226d36d ns: mdt-lustre-MDT0001_UUID lock: 00000000def62799/0x180b24bba140f0e0 lrc: 3/0,0 mode: CW/CW res: [0x2400007ed:0x141:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.202.22@tcp remote: 0x222e275a1cf19e6c expref: 4 pid: 9943 timeout: 0 lvb_type: 0 [ 6553.646264] Lustre: server umount lustre-MDT0001 complete [ 6553.654104] Lustre: Skipped 1 previous similar message [ 6577.544506] LDISKFS-fs (dm-1): recovery complete [ 6577.552154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6578.185070] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6578.203449] Lustre: Skipped 1 previous similar message [ 6579.515628] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6579.526468] Lustre: Skipped 1 previous similar message [ 6583.003898] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6586.547339] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:349 to 0x280000400:385 [ 6586.553406] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:155 to 0x2c0000400:193 [ 6596.519325] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6599.005233] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6623.373716] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 13:02:59 (1776186179) [ 6751.659796] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6764.158671] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 6766.624772] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 6766.627835] Lustre: Skipped 4 previous similar messages [ 6767.589115] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6767.612657] Lustre: Skipped 10 previous similar messages [ 6774.752253] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186324/real 1776186324] req@0000000024109516 x1862459342200256/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776186331 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 6774.797429] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6774.805337] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6774.836340] LustreError: Skipped 1 previous similar message [ 6791.035991] LDISKFS-fs (dm-0): recovery complete [ 6791.043219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6792.186277] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba140643f to 0x180b24bba145ceb6 [ 6792.209660] Lustre: Skipped 1 previous similar message [ 6798.839124] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6803.144276] Lustre: lustre-MDT0000: Recovery over after 0:11, of 2 clients 2 recovered and 0 were evicted. [ 6803.159167] Lustre: Skipped 2 previous similar messages [ 6803.236164] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5742 to 0x0:5761 [ 6803.238547] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6030 to 0x0:6049 [ 6808.991901] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6810.444952] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6915.976731] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 13:07:52 (1776186472) [ 7045.582174] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7057.631696] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 7059.774990] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.22@tcp (stopping) [ 7086.359206] LDISKFS-fs (dm-0): recovery complete [ 7086.363683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7087.734221] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.202.22@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7087.750717] LustreError: Skipped 135 previous similar messages [ 7106.487339] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7109.103213] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6020 to 0x0:6049 [ 7109.104155] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6309 to 0x0:6337 [ 7116.805237] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7118.597099] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7127.955182] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 13:11:24 (1776186684) [ 7258.281372] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7270.572811] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 7273.071896] Lustre: Failing over lustre-MDT0001 [ 7273.073783] Lustre: Skipped 2 previous similar messages [ 7273.083391] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 7273.100184] LustreError: Skipped 2 previous similar messages [ 7273.102142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7273.230362] Lustre: server umount lustre-MDT0001 complete [ 7273.232376] Lustre: Skipped 2 previous similar messages [ 7297.043922] LDISKFS-fs (dm-1): recovery complete [ 7297.052682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7297.388158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7297.395325] Lustre: Skipped 3 previous similar messages [ 7297.438740] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7297.441828] Lustre: Skipped 2 previous similar messages [ 7297.652734] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7297.654745] Lustre: Skipped 2 previous similar messages [ 7301.944919] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7302.637301] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 7302.649228] Lustre: Skipped 17 previous similar messages [ 7302.731720] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1027 to 0x280000400:1057 [ 7302.737590] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:835 to 0x2c0000400:865 [ 7311.694758] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7313.435758] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7322.536619] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 13:14:38 (1776186878) [ 7333.551142] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7335.585344] Lustre: lustre-OST0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnecting [ 7335.589225] Lustre: Skipped 1 previous similar message [ 7336.163390] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7365.373979] LDISKFS-fs (dm-2): recovery complete [ 7365.389192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7367.476694] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1027 to 0x280000400:1089 [ 7367.500710] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6918 to 0x0:6945 [ 7372.246281] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7383.410220] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7385.133898] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7397.876313] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 13:15:53 (1776186953) [ 7528.285380] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7536.058833] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7548.969536] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 7551.487459] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7551.508971] Lustre: Skipped 12 previous similar messages [ 7551.521960] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7555.817992] LustreError: 3352:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000002ff15b4 x1862459344848960/t0(0) o103->lustre-MDT0001-osp-MDT0000@0@lo:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'jbd2/dm-0-8.0' [ 7572.384164] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187122/real 1776187122] req@00000000f61d855a x1862459344849600/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776187129 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 7572.411139] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7572.415901] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7572.424051] LustreError: Skipped 1 previous similar message [ 7587.529810] LDISKFS-fs (dm-1): recovery complete [ 7587.535787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7689.229222] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7689.258585] LustreError: Skipped 161 previous similar messages [ 7695.357423] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7719.444356] LDISKFS-fs (dm-0): recovery complete [ 7719.447751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7726.053274] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba14cb548 to 0x180b24bba155185e [ 7726.067631] Lustre: Skipped 1 previous similar message [ 7732.239902] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7732.310374] Lustre: lustre-MDT0001: Recovery over after 0:41, of 2 clients 2 recovered and 0 were evicted. [ 7732.315402] Lustre: Skipped 3 previous similar messages [ 7732.383656] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:835 to 0x2c0000400:897 [ 7732.384632] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1027 to 0x280000400:1121 [ 7743.108317] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6918 to 0x0:6977 [ 7743.108407] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6631 to 0x0:6657 [ 7752.341322] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7755.496368] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7757.360451] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7766.572930] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 13:22:03 (1776187323) [ 7774.502120] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7805.037668] LDISKFS-fs (dm-0): recovery complete [ 7805.044807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7809.133335] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7813.311741] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7815.314511] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnected, waiting for 2 clients in recovery for 0:59 [ 7815.375922] Lustre: 195657:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000060706c03 x1862459334330880/t335007455270(335007455270) o101->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:48/0 lens 648/3424 e 0 to 0 dl 1776187378 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7815.455641] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6631 to 0x0:6689 [ 7815.465788] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6979 to 0x0:7009 [ 7822.941981] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7824.597972] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7834.271837] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 13:23:10 (1776187390) [ 7842.831229] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7869.431407] LDISKFS-fs (dm-0): recovery complete [ 7869.433476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7875.416520] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7875.421182] LustreError: 195657:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000007725603a x1862459334330880/t335007455270(335007455270) o101->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:108/0 lens 648/600 e 0 to 0 dl 1776187438 ref 1 fl Interpret:/4/0 rc 301/0 job:'lfs.0' [ 7879.221780] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7882.892357] Lustre: lustre-MDT0000: Client 00c2ea12-0d37-44cd-bf29-21c9c37834bb (at 192.168.202.22@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 7882.932651] Lustre: 195658:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000739d7655 x1862459334330880/t335007455270(335007455270) o101->00c2ea12-0d37-44cd-bf29-21c9c37834bb@192.168.202.22@tcp:116/0 lens 648/3424 e 0 to 0 dl 1776187446 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7883.080070] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7041 [ 7883.095919] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6631 to 0x0:6721 [ 7889.907262] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7891.499138] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7900.394759] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 13:24:16 (1776187456) [ 7904.760385] Lustre: Failing over lustre-OST0000 [ 7904.769063] Lustre: Skipped 5 previous similar messages [ 7904.927166] Lustre: server umount lustre-OST0000 complete [ 7904.930167] Lustre: Skipped 5 previous similar messages [ 7908.833234] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7908.835564] LustreError: Skipped 5 previous similar messages [ 7926.628894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.483640] Lustre: MGC192.168.202.122@tcp: Connection restored to (at 0@lo) [ 7935.502394] Lustre: Skipped 21 previous similar messages [ 7935.875675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7935.878848] Lustre: Skipped 5 previous similar messages [ 7935.934640] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7935.940414] Lustre: Skipped 5 previous similar messages [ 7941.092614] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 7941.104699] Lustre: Skipped 5 previous similar messages [ 7941.179281] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6631 to 0x0:6753 [ 7941.474217] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7951.444333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7953.093347] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1027 to 0x280000400:1153 [ 7953.107413] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7073 [ 7957.061873] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7966.692666] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:25:23 (1776187523) [ 7967.847511] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7967.853329] LustreError: 195661:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000e94fda69 x1862459345100096/t352187318279(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:201/0 lens 1056/4320 e 0 to 0 dl 1776187531 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7974.783429] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7974.890345] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8001.074494] LDISKFS-fs (dm-0): recovery complete [ 8001.087475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8010.154288] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8010.385821] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7105 [ 8010.387069] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6785 [ 8020.786075] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8022.376077] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8030.728563] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:26:27 (1776187587) [ 8031.917977] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8031.919863] LustreError: 195662:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000008ce6f61e x1862459345122816/t356482285578(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:265/0 lens 2200/4320 e 0 to 0 dl 1776187595 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 8039.170165] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8039.409384] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8045.768415] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8072.844072] LDISKFS-fs (dm-1): recovery complete [ 8072.846900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8077.162535] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8078.363350] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:908 to 0x2c0000400:929 [ 8078.364880] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1164 to 0x280000400:1185 [ 8085.189772] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8086.702770] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8095.925308] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:27:32 (1776187652) [ 8097.134492] LustreError: 204378:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000f564e81b x1862459345142720/t356482285597(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:330/0 lens 2200/4320 e 0 to 0 dl 1776187660 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 8104.060342] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8104.441136] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8110.707046] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8135.057731] LDISKFS-fs (dm-0): recovery complete [ 8135.060771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8142.940170] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8144.223923] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7137 [ 8144.229916] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6817 [ 8153.879577] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8155.582723] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8164.321191] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8164.340629] Lustre: Skipped 33 previous similar messages [ 8181.129099] LDISKFS-fs (dm-1): recovery complete [ 8181.133722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.948163] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8186.910743] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:940 to 0x2c0000400:961 [ 8186.911801] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1196 to 0x280000400:1217 [ 8194.444397] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8196.267463] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8205.210353] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:29:21 (1776187761) [ 8206.366609] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8206.374149] Lustre: Skipped 1 previous similar message [ 8213.472387] Lustre: 211545:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187763/real 1776187763] req@00000000b8c7abbc x1862459345180928/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2200/4320 e 0 to 1 dl 1776187770 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'osp_up0-1.0' [ 8213.512460] Lustre: 211545:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 8213.520070] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8216.340337] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8222.916961] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8228.326300] LustreError: 6239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187785 with bad export cookie 1732518869938320428 [ 8228.327800] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8228.333028] LustreError: 6239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8228.351750] LustreError: Skipped 5 previous similar messages [ 8250.361180] LDISKFS-fs (dm-0): recovery complete [ 8250.364568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8267.049976] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8289.248988] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8289.254434] LustreError: Skipped 305 previous similar messages [ 8290.567799] LDISKFS-fs (dm-1): recovery complete [ 8290.570316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8294.598217] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8296.840951] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6849 [ 8296.843963] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7169 [ 8296.939325] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1228 to 0x280000400:1249 [ 8296.939712] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:972 to 0x2c0000400:993 [ 8305.852394] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8308.294479] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8310.296366] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8320.267291] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:31:16 (1776187876) [ 8321.306226] LustreError: 214219:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000009f66a556 x1862459335336640/t30064771142(0) o36->1e0dbe72-1f67-4603-9d72-6c159f8e83c7@192.168.202.22@tcp:554/0 lens 560/448 e 0 to 0 dl 1776187884 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 8321.331382] LustreError: 214219:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 8328.314400] Lustre: lustre-MDT0001: Client 1e0dbe72-1f67-4603-9d72-6c159f8e83c7 (at 192.168.202.22@tcp) reconnecting [ 8328.329082] Lustre: 215610:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000d1ed1ece x1862459335336640/t30064771142(0) o36->1e0dbe72-1f67-4603-9d72-6c159f8e83c7@192.168.202.22@tcp:561/0 lens 560/448 e 0 to 0 dl 1776187891 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8330.832686] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8354.406271] LDISKFS-fs (dm-0): recovery complete [ 8354.409906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.441727] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba155c3c2 to 0x180b24bba155d201 [ 8361.445788] Lustre: Skipped 6 previous similar messages [ 8365.870401] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8367.159815] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8367.168471] Lustre: Skipped 11 previous similar messages [ 8367.210959] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7201 [ 8367.211597] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6881 [ 8374.446944] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8375.991380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8384.216890] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:32:20 (1776187940) [ 8392.472573] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8392.822857] Lustre: lustre-MDT0001: Client 1e0dbe72-1f67-4603-9d72-6c159f8e83c7 (at 192.168.202.22@tcp) reconnecting [ 8392.862611] Lustre: 214218:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000033ac33d1 x1862459335354048/t30064771212(0) o36->1e0dbe72-1f67-4603-9d72-6c159f8e83c7@192.168.202.22@tcp:626/0 lens 560/448 e 0 to 0 dl 1776187956 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8416.297432] LDISKFS-fs (dm-1): recovery complete [ 8416.310267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8421.097941] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8421.942629] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1014 to 0x2c0000400:1057 [ 8421.945687] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1270 to 0x280000400:1313 [ 8429.870844] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8431.330158] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8438.928020] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:33:15 (1776187995) [ 8447.092763] Lustre: lustre-MDT0001: Client 1e0dbe72-1f67-4603-9d72-6c159f8e83c7 (at 192.168.202.22@tcp) reconnecting [ 8447.119710] Lustre: 214219:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000053a4e71c x1862459335369984/t34359738438(0) o36->1e0dbe72-1f67-4603-9d72-6c159f8e83c7@192.168.202.22@tcp:680/0 lens 560/448 e 0 to 0 dl 1776188010 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8449.559589] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8456.091992] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8479.035881] LDISKFS-fs (dm-0): recovery complete [ 8479.041264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8491.851973] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8493.154128] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6913 [ 8493.155356] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7233 [ 8500.905810] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8502.471630] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8506.569489] Lustre: Failing over lustre-MDT0001 [ 8506.576784] Lustre: Skipped 10 previous similar messages [ 8506.940783] Lustre: server umount lustre-MDT0001 complete [ 8506.945714] Lustre: Skipped 10 previous similar messages [ 8530.488935] LDISKFS-fs (dm-1): recovery complete [ 8530.492932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8535.241579] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8536.037979] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 8536.041420] Lustre: Skipped 45 previous similar messages [ 8536.106704] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1068 to 0x2c0000400:1089 [ 8536.110373] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1324 to 0x280000400:1345 [ 8546.305125] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8548.188310] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8557.448658] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:35:14 (1776188114) [ 8558.308622] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8558.312745] Lustre: Skipped 3 previous similar messages [ 8565.363526] Lustre: lustre-MDT0001: Client 1e0dbe72-1f67-4603-9d72-6c159f8e83c7 (at 192.168.202.22@tcp) reconnecting [ 8565.389488] Lustre: 214218:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000005687ac5c x1862459335392384/t38654705734(0) o36->1e0dbe72-1f67-4603-9d72-6c159f8e83c7@192.168.202.22@tcp:43/0 lens 560/448 e 0 to 0 dl 1776188128 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8568.843682] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8577.174645] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8583.701430] LustreError: 9221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188140 with bad export cookie 1732518869938334106 [ 8583.707421] LustreError: 9221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8604.337337] LDISKFS-fs (dm-0): recovery complete [ 8604.342407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8616.126913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8616.131329] Lustre: Skipped 11 previous similar messages [ 8616.167566] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8616.176347] Lustre: Skipped 11 previous similar messages [ 8620.359916] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8633.973901] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8633.981687] Lustre: Skipped 11 previous similar messages [ 8641.003568] LDISKFS-fs (dm-1): recovery complete [ 8641.008096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8641.225893] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 8641.233509] LustreError: Skipped 8 previous similar messages [ 8645.558794] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8647.092501] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6945 [ 8647.095392] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7265 [ 8647.151989] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1100 to 0x2c0000400:1121 [ 8647.156872] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1356 to 0x280000400:1377 [ 8654.610941] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8656.144561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8657.568666] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8666.374292] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:37:03 (1776188223) [ 8667.435252] LustreError: 185248:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000005687e901 x1862459345321920/t377957122063(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:145/0 lens 640/4320 e 0 to 0 dl 1776188230 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 8667.463091] LustreError: 185248:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 3 previous similar messages [ 8672.980639] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8674.463459] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.22@tcp (stopping) [ 8674.466743] Lustre: Skipped 4 previous similar messages [ 8679.905872] LustreError: 228502:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b3afe4fa x1862459345325056/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/0/ffffffff rc 0/-1 job:'umount.0' [ 8679.932534] LustreError: 228502:0:(osp_object.c:629:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8701.788679] LDISKFS-fs (dm-1): recovery complete [ 8701.791743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8702.025660] LustreError: 229191:0:(llog_cat.c:407:llog_cat_id2handle()) lustre-MDT0000-osp-MDT0001: error opening log id [0x20002abb1:0x1:0x0]: rc = -2 [ 8705.576352] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8707.146547] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1409 [ 8707.149094] Lustre: 226044:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000830d7d02 x1862459335412480/t42949673032(0) o36->1e0dbe72-1f67-4603-9d72-6c159f8e83c7@192.168.202.22@tcp:210/0 lens 496/2888 e 0 to 0 dl 1776188295 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0' [ 8707.163469] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1132 to 0x2c0000400:1153 [ 8713.810858] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8715.204558] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8722.370781] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:37:59 (1776188279) [ 8728.871679] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8754.660158] LDISKFS-fs (dm-0): recovery complete [ 8754.671332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8763.492320] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8765.021852] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:6977 [ 8765.023947] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7297 [ 8772.091757] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8773.580798] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8780.411802] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:38:57 (1776188337) [ 8788.448306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.455825] Lustre: Skipped 32 previous similar messages [ 8788.467144] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8789.034285] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8795.104886] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8817.764316] LDISKFS-fs (dm-0): recovery complete [ 8817.769839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8822.245362] LustreError: 233736:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.202.122@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000000fc4c0bb) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 8825.997852] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8828.000555] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7009 [ 8828.009162] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7329 [ 8836.177353] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8837.956289] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8866.115029] LDISKFS-fs (dm-1): recovery complete [ 8866.119503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8871.938495] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8872.002134] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1132 to 0x2c0000400:1185 [ 8872.007632] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1441 [ 8883.179781] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8885.130801] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8894.128857] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:40:50 (1776188450) [ 8902.625483] Lustre: 235184:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776188452/real 1776188452] req@00000000b9d0faec x1862459345394688/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1728/4320 e 0 to 1 dl 1776188459 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'osp_up0-1.0' [ 8902.658452] Lustre: 235184:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 28 previous similar messages [ 8902.685974] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8904.365691] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8913.086845] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8917.990976] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8918.007762] LustreError: Skipped 264 previous similar messages [ 8919.343558] LustreError: 9221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188476 with bad export cookie 1732518869938344438 [ 8919.350847] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8919.356676] LustreError: Skipped 5 previous similar messages [ 8919.691743] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.22@tcp (stopping) [ 8919.701804] Lustre: Skipped 7 previous similar messages [ 8952.264446] LDISKFS-fs (dm-0): recovery complete [ 8952.267581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8973.345850] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba15611f6 to 0x180b24bba1561b9d [ 8973.352646] Lustre: Skipped 4 previous similar messages [ 8977.939097] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9000.769510] LDISKFS-fs (dm-1): recovery complete [ 9000.774079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9005.551908] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9006.241312] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9006.253971] Lustre: Skipped 9 previous similar messages [ 9006.283789] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1132 to 0x2c0000400:1217 [ 9006.287553] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1473 [ 9006.471928] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7041 [ 9006.473657] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7361 [ 9014.307620] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9015.447741] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9016.703612] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9025.767943] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:43:01 (1776188581) [ 9034.359129] Lustre: lustre-MDT0001: Client 1e0dbe72-1f67-4603-9d72-6c159f8e83c7 (at 192.168.202.22@tcp) reconnecting [ 9034.385614] Lustre: 237860:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000e378cbf5 x1862459335449408/t55834574850(0) o36->1e0dbe72-1f67-4603-9d72-6c159f8e83c7@192.168.202.22@tcp:512/0 lens 496/2888 e 0 to 0 dl 1776188597 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0' [ 9036.417220] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9063.292559] LDISKFS-fs (dm-0): recovery complete [ 9063.295487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9072.277901] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7393 [ 9072.291878] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7073 [ 9074.164172] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9085.580073] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9087.146797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9096.272111] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:44:12 (1776188652) [ 9097.472247] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 9097.474686] Lustre: Skipped 5 previous similar messages [ 9105.531966] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9107.722789] Lustre: Failing over lustre-MDT0001 [ 9107.729488] Lustre: Skipped 9 previous similar messages [ 9107.936828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9107.947135] Lustre: Skipped 9 previous similar messages [ 9108.032315] Lustre: server umount lustre-MDT0001 complete [ 9108.034695] Lustre: Skipped 9 previous similar messages [ 9131.615666] LDISKFS-fs (dm-1): recovery complete [ 9131.621494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9136.156514] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9137.123238] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 9137.134721] Lustre: Skipped 39 previous similar messages [ 9137.199679] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1505 [ 9137.200625] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1132 to 0x2c0000400:1249 [ 9146.047281] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9147.789945] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9156.084508] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:45:12 (1776188712) [ 9164.778565] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9171.483030] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9198.086392] LDISKFS-fs (dm-0): recovery complete [ 9198.090305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9206.933614] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9207.516988] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7425 [ 9207.520473] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7105 [ 9216.316681] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9218.161371] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9245.575098] LDISKFS-fs (dm-1): recovery complete [ 9245.579623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9245.972034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9245.981847] Lustre: Skipped 10 previous similar messages [ 9246.021465] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 9246.033578] Lustre: Skipped 10 previous similar messages [ 9247.868553] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9247.875740] Lustre: Skipped 10 previous similar messages [ 9250.650166] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9251.371857] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1132 to 0x2c0000400:1281 [ 9251.373379] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1537 [ 9260.025268] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9261.689727] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9271.440486] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:47:08 (1776188828) [ 9273.306563] LustreError: 237860:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000e52ca7e5 x1862459335478336/t64424509442(0) o36->1e0dbe72-1f67-4603-9d72-6c159f8e83c7@192.168.202.22@tcp:751/0 lens 496/456 e 0 to 0 dl 1776188836 ref 1 fl Interpret:/0/0 rc 0/0 job:'rmdir.0' [ 9273.330120] LustreError: 237860:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 6 previous similar messages [ 9281.717124] Lustre: lustre-MDT0001: Client 1e0dbe72-1f67-4603-9d72-6c159f8e83c7 (at 192.168.202.22@tcp) reconnecting [ 9281.734578] Lustre: Skipped 2 previous similar messages [ 9282.262719] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9289.225433] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9293.812037] LustreError: 182361:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188851 with bad export cookie 1732518869938351970 [ 9293.826304] LustreError: 182361:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9316.479210] LDISKFS-fs (dm-0): recovery complete [ 9316.485785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9331.241265] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9353.558539] LDISKFS-fs (dm-1): recovery complete [ 9353.560357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.677706] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 9353.680178] LustreError: Skipped 7 previous similar messages [ 9358.305151] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9359.019512] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1569 [ 9359.028184] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1132 to 0x2c0000400:1313 [ 9359.282174] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6755 to 0x0:7137 [ 9359.284340] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7011 to 0x0:7457 [ 9370.087749] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9371.874123] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9373.733175] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9383.095494] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 13:48:59 (1776188939) [ 9385.398874] Lustre: 251740:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 1e0dbe72-1f67-4603-9d72-6c159f8e83c7 at adminstrative request [ 9394.294198] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:49:10 (1776188950) [ 9400.295822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9400.308155] Lustre: Skipped 34 previous similar messages [ 9419.929044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9434.034612] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9434.852443] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7508 to 0x0:7553 [ 9434.854463] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7189 to 0x0:7233 [ 9442.965039] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9444.469805] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9452.488444] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:50:09 (1776189009) [ 9484.618116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9486.568955] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1601 [ 9486.573948] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7654 to 0x0:7681 [ 9488.526078] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9496.575773] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9498.069693] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9506.583952] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 13:51:03 (1776189063) [ 9518.896666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9519.054480] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9519.067960] LustreError: Skipped 223 previous similar messages [ 9523.852579] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9524.792598] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7654 to 0x0:7713 [ 9524.793059] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7189 to 0x0:7265 [ 9532.626888] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 13:51:29 (1776189089) [ 9541.053531] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9564.995662] LDISKFS-fs (dm-2): recovery complete [ 9564.999791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9567.286556] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7715 to 0x0:7745 [ 9567.292650] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1633 [ 9568.661048] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9576.643264] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9578.040531] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9585.412119] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 13:52:22 (1776189142) [ 9592.923307] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9620.150102] LDISKFS-fs (dm-2): recovery complete [ 9620.156636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9622.283243] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.22@tcp inode [0x20002f201:0x5:0x0] object 0x0:7746 extent [0-4194303]: client csum 1e094c4d, server csum f826991b [ 9622.374396] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9622.375416] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7747 to 0x0:7777 [ 9622.379122] Lustre: Skipped 11 previous similar messages [ 9622.386937] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1665 [ 9624.003849] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9631.965102] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9633.505453] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9640.890756] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 13:53:17 (1776189197) [ 9648.123377] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9655.891701] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9665.209091] LustreError: 6239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189222 with bad export cookie 1732518869938378647 [ 9665.220199] LustreError: 166-1: MGC192.168.202.122@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9665.223042] LustreError: 6239:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9665.254602] LustreError: Skipped 5 previous similar messages [ 9688.777704] LDISKFS-fs (dm-0): recovery complete [ 9688.785181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9698.294549] Lustre: Evicted from MGS (at 192.168.202.122@tcp) after server handle changed from 0x180b24bba1569797 to 0x180b24bba156a9a3 [ 9698.311680] Lustre: Skipped 5 previous similar messages [ 9703.559201] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9704.695914] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7189 to 0x0:7297 [ 9725.627197] LDISKFS-fs (dm-2): recovery complete [ 9725.634674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9727.786216] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1697 [ 9729.995311] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9743.319088] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 13:54:59 (1776189299) [ 9752.356897] Lustre: Failing over lustre-OST0000 [ 9752.359829] Lustre: Skipped 11 previous similar messages [ 9752.388429] Lustre: server umount lustre-OST0000 complete [ 9752.390013] Lustre: Skipped 11 previous similar messages [ 9763.744167] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776189313/real 1776189313] req@0000000046ccc07b x1862459345642240/t0(0) o400->MGC192.168.202.122@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776189320 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 9763.767458] Lustre: 3355:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 12 previous similar messages [ 9774.488449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9781.222030] Lustre: MGC192.168.202.122@tcp: Connection restored to 192.168.202.122@tcp (at 0@lo) [ 9781.233034] Lustre: Skipped 39 previous similar messages [ 9785.444361] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9786.935282] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7189 to 0x0:7329 [ 9793.800450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9800.846945] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9803.740178] Lustre: lustre-OST0000: Denying connection for new client 988c6ad9-cfbf-47e9-8bb2-5be7fd8d5128 (at 192.168.202.22@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 9803.756626] Lustre: Skipped 20 previous similar messages [ 9824.386254] Lustre: lustre-OST0000: Denying connection for new client 988c6ad9-cfbf-47e9-8bb2-5be7fd8d5128 (at 192.168.202.22@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:41 [ 9824.407103] Lustre: Skipped 3 previous similar messages [ 9860.222351] Lustre: lustre-OST0000: Denying connection for new client 988c6ad9-cfbf-47e9-8bb2-5be7fd8d5128 (at 192.168.202.22@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:05 [ 9860.237467] Lustre: Skipped 6 previous similar messages [ 9866.000253] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9866.006888] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9866.047249] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7828 to 0x0:7849 [ 9866.051617] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1729 [ 9872.755946] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 59 sec [ 9882.819146] Lustre: DEBUG MARKER: free_before: 7646580 free_after: 7646580 [ 9889.569372] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 13:57:25 (1776189445) [ 9915.979437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9916.224828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9916.234809] Lustre: Skipped 11 previous similar messages [ 9916.245113] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 9916.253165] Lustre: Skipped 11 previous similar messages [ 9916.546285] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9916.556785] Lustre: Skipped 11 previous similar messages [ 9917.695811] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7852 to 0x0:7881 [ 9917.709918] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1761 [ 9921.066772] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9933.696549] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 13:58:09 (1776189489) [ 9958.908633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9960.774878] LustreError: 268411:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9960.783882] LustreError: 268411:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 9961.632479] Lustre: *** cfs_fail_loc=715, val=0*** [ 9965.795386] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9967.073204] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:58 [ 9967.104007] Lustre: Skipped 1 previous similar message [ 9968.109092] Lustre: *** cfs_fail_loc=715, val=0*** [ 9968.110921] Lustre: Skipped 2 previous similar messages [ 9974.245668] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:51 [ 9974.257091] Lustre: Skipped 2 previous similar messages [ 9975.265667] Lustre: *** cfs_fail_loc=715, val=0*** [ 9975.276291] Lustre: Skipped 2 previous similar messages [ 9981.410546] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:44 [ 9981.430452] Lustre: Skipped 2 previous similar messages [ 9982.432327] Lustre: *** cfs_fail_loc=715, val=0*** [ 9982.433977] Lustre: Skipped 2 previous similar messages [ 9988.588289] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:37 [ 9988.598226] Lustre: Skipped 2 previous similar messages [ 9989.600820] Lustre: *** cfs_fail_loc=715, val=0*** [ 9989.604332] Lustre: Skipped 2 previous similar messages [ 9995.745490] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:30 [ 9995.752265] Lustre: Skipped 2 previous similar messages [10000.808135] LustreError: 268411:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [10000.812271] LustreError: 268411:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [10000.855015] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1388 to 0x280000400:1793 [10000.855487] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7883 to 0x0:7913 [10007.663737] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10009.439279] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10018.772300] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 13:59:35 (1776189575) [10025.955413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10025.973207] Lustre: Skipped 29 previous similar messages [10043.939636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10052.773548] Lustre: *** cfs_fail_loc=715, val=0*** [10052.781752] Lustre: Skipped 5 previous similar messages [10056.693201] LustreError: 269978:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [10057.537664] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [10059.379624] Lustre: lustre-MDT0000: Client 988c6ad9-cfbf-47e9-8bb2-5be7fd8d5128 (at 192.168.202.22@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [10059.405976] Lustre: Skipped 1 previous similar message [10062.816769] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10069.985526] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10071.009423] Lustre: *** cfs_fail_loc=715, val=0*** [10071.016200] Lustre: Skipped 5 previous similar messages [10077.156556] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10080.884618] Lustre: lustre-MDT0000: Client 988c6ad9-cfbf-47e9-8bb2-5be7fd8d5128 (at 192.168.202.22@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [10080.913474] Lustre: Skipped 2 previous similar messages [10091.488480] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10091.499863] Lustre: Skipped 1 previous similar message [10105.825827] Lustre: *** cfs_fail_loc=715, val=0*** [10105.830724] Lustre: Skipped 9 previous similar messages [10111.968640] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10111.982258] Lustre: Skipped 2 previous similar messages [10115.698700] Lustre: lustre-MDT0000: Client 988c6ad9-cfbf-47e9-8bb2-5be7fd8d5128 (at 192.168.202.22@tcp) reconnected, waiting for 2 clients in recovery for 0:02 [10115.715462] Lustre: Skipped 4 previous similar messages [10122.868788] Lustre: lustre-MDT0000: Recovery already passed deadline 0:04. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [10130.035640] Lustre: lustre-MDT0000: Recovery already passed deadline 0:12. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [10136.720134] LustreError: 269978:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [10136.742909] Lustre: 269978:0:(ldlm_lib.c:2849:target_recovery_thread()) too long recovery - read logs [10136.762748] LustreError: dumping log to /tmp/lustre-log.1776189693.269978 [10136.881980] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7342 to 0x0:7361 [10136.883545] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7924 to 0x0:7945 [10144.393680] Lustre: DEBUG MARKER: oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10146.339694] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10155.455716] Lustre: DEBUG MARKER: == replay-single test complete, duration 9904 sec ======== 14:01:51 (1776189711) [10167.777730] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10167.784162] LustreError: Skipped 17 previous similar messages [10167.824405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10169.322185] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10169.351894] LustreError: Skipped 226 previous similar messages [10218.942600] Lustre: DEBUG MARKER: oleg222-server.virtnet: executing unload_modules_local [10222.855796] Key type lgssc unregistered [10223.265760] LNet: 273173:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10223.289105] LNet: Removed LNI 192.168.202.122@tcp [10224.543649] Key type .llcrypt unregistered [10224.546029] Key type ._llcrypt unregistered