[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 472303424 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002398] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.010040] LSM: Security Framework initializing [ 0.011046] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.014046] *** VALIDATE selinux *** [ 0.022277] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026759] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030073] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] *** VALIDATE tmpfs *** [ 0.033039] *** VALIDATE proc *** [ 0.034280] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.036267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039037] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.043805] debug: unmapping init [mem 0xffffffffb8659000-0xffffffffb8660fff] [ 0.045215] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047023] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054317] rcu: Hierarchical SRCU implementation. [ 0.056582] smp: Bringing up secondary CPUs ... [ 0.057624] x86: Booting SMP configuration: [ 0.058034] .... node #0, CPUs: #1 #2 #3 [ 0.062478] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144234] node 0 deferred pages initialised in 77ms [ 0.148128] devtmpfs: initialized [ 0.149269] x86/mm: Memory block size: 128MB [ 0.152000] gcov: version magic: 0x41383552 [ 0.154222] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161306] pinctrl core: initialized pinctrl subsystem [ 0.163158] [ 0.163668] ************************************************************* [ 0.165013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167014] ** ** [ 0.169012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171013] ** ** [ 0.173062] ** This means that this kernel is built to expose internal ** [ 0.175011] ** IOMMU data structures, which may compromise security on ** [ 0.177012] ** your system. ** [ 0.179012] ** ** [ 0.181012] ** If you see this message and you are not debugging the ** [ 0.183012] ** kernel, report this immediately to your vendor! ** [ 0.185011] ** ** [ 0.188014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190012] ************************************************************* [ 0.191571] NET: Registered protocol family 16 [ 0.193453] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202028] cpuidle: using governor menu [ 0.203711] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205378] PCI: Using configuration type 1 for base access [ 0.209137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217109] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.218026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220054] cryptd: max_cpu_qlen set to 1000 [ 0.222168] ACPI: Added _OSI(Module Device) [ 0.223017] ACPI: Added _OSI(Processor Device) [ 0.225017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.231000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237353] ACPI: Interpreter enabled [ 0.239068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240013] ACPI: Using IOAPIC for interrupt routing [ 0.242222] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245418] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.255474] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266083] acpiphp: Slot [2] registered [ 0.268126] acpiphp: Slot [5] registered [ 0.269050] acpiphp: Slot [6] registered [ 0.270113] acpiphp: Slot [7] registered [ 0.270966] acpiphp: Slot [8] registered [ 0.272082] acpiphp: Slot [9] registered [ 0.273112] acpiphp: Slot [10] registered [ 0.274085] acpiphp: Slot [3] registered [ 0.275014] acpiphp: Slot [4] registered [ 0.276091] acpiphp: Slot [11] registered [ 0.278114] acpiphp: Slot [12] registered [ 0.279114] acpiphp: Slot [13] registered [ 0.281169] acpiphp: Slot [14] registered [ 0.283118] acpiphp: Slot [15] registered [ 0.285238] acpiphp: Slot [16] registered [ 0.287100] acpiphp: Slot [17] registered [ 0.289114] acpiphp: Slot [18] registered [ 0.290113] acpiphp: Slot [19] registered [ 0.292105] acpiphp: Slot [20] registered [ 0.294171] acpiphp: Slot [21] registered [ 0.295191] acpiphp: Slot [22] registered [ 0.297157] acpiphp: Slot [23] registered [ 0.299136] acpiphp: Slot [24] registered [ 0.301088] acpiphp: Slot [25] registered [ 0.302095] acpiphp: Slot [26] registered [ 0.304109] acpiphp: Slot [27] registered [ 0.305112] acpiphp: Slot [28] registered [ 0.307135] acpiphp: Slot [29] registered [ 0.309147] acpiphp: Slot [30] registered [ 0.310130] acpiphp: Slot [31] registered [ 0.312088] PCI host bridge to bus 0000:00 [ 0.313021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.316026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.319025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.321026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.324023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.326030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.328183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.330896] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.334091] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344548] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.349064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.351019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.355018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.357015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.359606] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.362800] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.366048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.368892] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.374021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.387021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.392019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.400361] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.440177] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.454018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.474035] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.484946] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.490019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.497017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.514018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.524928] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.532016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.539018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.555018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.563890] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.570017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.577018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.594022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.605009] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.611015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.616020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.633018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.644245] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.646397] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.649737] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.651448] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.654241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.659153] iommu: Default domain type: Passthrough [ 0.661496] SCSI subsystem initialized [ 0.664146] ACPI: bus type USB registered [ 0.665130] usbcore: registered new interface driver usbfs [ 0.668096] usbcore: registered new interface driver hub [ 0.671092] usbcore: registered new device driver usb [ 0.673222] pps_core: LinuxPPS API ver. 1 registered [ 0.676015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.681054] PTP clock support registered [ 0.683055] EDAC MC: Ver: 3.0.0 [ 0.685121] PCI: Using ACPI for IRQ routing [ 0.686971] NetLabel: Initializing [ 0.689300] NetLabel: domain hash size = 128 [ 0.690011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.692083] NetLabel: unlabeled traffic allowed by default [ 0.694144] vgaarb: loaded [ 0.696273] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.698018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.705022] clocksource: Switched to clocksource kvm-clock [ 0.815997] VFS: Disk quotas dquot_6.6.0 [ 0.817898] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.821048] *** VALIDATE ramfs *** [ 0.822717] *** VALIDATE hugetlbfs *** [ 0.824501] pnp: PnP ACPI init [ 0.827162] pnp: PnP ACPI: found 6 devices [ 0.859158] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.863410] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.866025] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.868568] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.871532] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.875513] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.879015] NET: Registered protocol family 2 [ 0.882518] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.888427] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.892817] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.899089] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.903305] TCP: Hash tables configured (established 65536 bind 65536) [ 0.906349] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.909668] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.914111] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.917472] NET: Registered protocol family 1 [ 0.920429] RPC: Registered named UNIX socket transport module. [ 0.923032] RPC: Registered udp transport module. [ 0.924713] RPC: Registered tcp transport module. [ 0.926686] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.928923] NET: Registered protocol family 44 [ 0.932024] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.934104] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.936314] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.938943] PCI: CLS 0 bytes, default 64 [ 0.940747] Unpacking initramfs... [ 2.348674] debug: unmapping init [mem 0xffff8dcffcc54000-0xffff8dcffffbffff] [ 2.352515] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.354946] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.357491] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.876080] Initialise system trusted keyrings [ 2.878154] Key type blacklist registered [ 2.880600] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.891415] zbud: loaded [ 2.895474] *** VALIDATE nfs *** [ 2.897273] *** VALIDATE nfs4 *** [ 2.899550] pstore: using deflate compression [ 2.902929] Platform Keyring initialized [ 3.031873] NET: Registered protocol family 38 [ 3.035417] Key type asymmetric registered [ 3.037078] Asymmetric key parser 'x509' registered [ 3.039018] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.042331] io scheduler mq-deadline registered [ 3.044344] io scheduler kyber registered [ 3.046161] io scheduler bfq registered [ 3.048956] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.052122] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.056529] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.060362] ACPI: Power Button [PWRF] [ 3.067363] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.075094] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.086364] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.092978] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.112740] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.141312] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.169496] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.175920] Non-volatile memory driver v1.3 [ 3.177894] Linux agpgart interface v0.103 [ 3.211777] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.215265] vda: detected capacity change from 0 to 34836480 [ 3.229120] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.231947] vdb: detected capacity change from 0 to 1073741824 [ 3.251508] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.255258] vdc: detected capacity change from 0 to 2621440000 [ 3.276620] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.280141] vdd: detected capacity change from 0 to 2621440000 [ 3.297030] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.300844] vde: detected capacity change from 0 to 4294967296 [ 3.316318] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.320809] vdf: detected capacity change from 0 to 4294967296 [ 3.327992] libphy: Fixed MDIO Bus: probed [ 3.333367] usbcore: registered new interface driver usbserial_generic [ 3.338040] usbserial: USB Serial support registered for generic [ 3.340759] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.345692] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.347681] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.350402] mousedev: PS/2 mouse device common for all mice [ 3.353866] rtc_cmos 00:05: RTC can wake from S4 [ 3.357275] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.361641] rtc_cmos 00:05: registered as rtc0 [ 3.364724] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.366224] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.373736] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.374282] intel_pstate: CPU model not supported [ 3.382089] hid: raw HID events driver (C) Jiri Kosina [ 3.384472] usbcore: registered new interface driver usbhid [ 3.386823] usbhid: USB HID core driver [ 3.388796] drop_monitor: Initializing network drop monitor service [ 3.391451] Initializing XFRM netlink socket [ 3.393884] NET: Registered protocol family 10 [ 3.397873] Segment Routing with IPv6 [ 3.399374] NET: Registered protocol family 17 [ 3.401626] mpls_gso: MPLS GSO support [ 3.408186] RAS: Correctable Errors collector initialized. [ 3.410309] AVX version of gcm_enc/dec engaged. [ 3.411993] AES CTR mode by8 optimization enabled [ 3.497215] sched_clock: Marking stable (3497192481, 0)->(4427894705, -930702224) [ 3.501199] registered taskstats version 1 [ 3.503425] Loading compiled-in X.509 certificates [ 3.505702] zswap: loaded using pool lzo/zbud [ 3.534774] Key type big_key registered [ 3.547211] Key type encrypted registered [ 3.548987] ima: No TPM chip found, activating TPM-bypass! [ 3.551078] ima: Allocated hash algorithm: sha1 [ 3.552829] ima: No architecture policies found [ 3.554630] evm: Initialising EVM extended attributes: [ 3.556418] evm: security.selinux [ 3.557770] evm: security.ima [ 3.558884] evm: security.capability [ 3.560281] evm: HMAC attrs: 0x1 [ 3.562832] rtc_cmos 00:05: setting system clock to 2026-04-14 15:31:44 UTC (1776180704) [ 3.569495] debug: unmapping init [mem 0xffffffffb9603000-0xffffffffb97fffff] [ 3.573030] debug: unmapping init [mem 0xffffffffb8382000-0xffffffffb8658fff] [ 3.583088] Write protecting the kernel read-only data: 28672k [ 3.586320] debug: unmapping init [mem 0xffffffffb6a03000-0xffffffffb6bfffff] [ 3.588746] debug: unmapping init [mem 0xffffffffb7314000-0xffffffffb73fffff] [ 3.622155] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.630897] systemd[1]: Detected virtualization kvm. [ 3.632278] systemd[1]: Detected architecture x86-64. [ 3.634458] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.661502] systemd[1]: No hostname configured. [ 3.663448] systemd[1]: Set hostname to . [ 3.665558] random: systemd: uninitialized urandom read (16 bytes read) [ 3.668431] systemd[1]: Initializing machine ID from random generator. [ 3.816837] random: systemd: uninitialized urandom read (16 bytes read) [ 3.820874] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.825480] random: systemd: uninitialized urandom read (16 bytes read) [ 3.828451] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.834560] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.507043] device-mapper: uevent: version 1.0.3 [ 4.509630] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.286047] virtio_net virtio0 ens2: renamed from eth0 [ 5.297623] scsi host0: ata_piix [ 5.306297] random: fast init done [ 5.335832] scsi host1: ata_piix [ 5.338732] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.341193] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.878934] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.071486] random: crng init done [ 10.073046] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.612266] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.723742] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.005235] SELinux: Disabled at runtime. [ 12.067814] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.076427] systemd[1]: Detected virtualization kvm. [ 12.078516] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.589297] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.592773] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.598311] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.602484] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.605612] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.613613] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.619374] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.783505] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.041966] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.312965] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.319773] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.420633] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.435742] EDAC sbridge: Ver: 1.1.2 [ 15.189579] Key type dns_resolver registered [ 15.490808] NFS: Registering the id_resolver key type [ 15.492803] Key type id_resolver registered [ 15.493912] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg434-server login: [ 36.678677] libcfs: loading out-of-tree module taints kernel. [ 36.699277] alg: No test for adler32 (adler32-zlib) [ 37.451429] Key type ._llcrypt registered [ 37.452852] Key type .llcrypt registered [ 37.513595] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_hostid [ 52.622721] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 53.817293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.617903] Lustre: Lustre: Build Version: 2.15.8_13_g05ba8a9 [ 55.361238] LNet: Added LNI 192.168.204.134@tcp [8/256/0/180] [ 55.367069] LNet: Accept secure, port 988 [ 57.079899] Key type lgssc registered [ 58.510906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.723243] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.372950] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.713085] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.288571] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 97.233238] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.336310] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing load_modules_local [ 116.883357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.938294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 118.092376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 118.138895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 118.231543] Lustre: lustre-MDT0000: new disk, initializing [ 118.310041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.330755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 121.749495] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 132.618306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 132.712612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 132.795820] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 132.830837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 132.833624] Lustre: Skipped 1 previous similar message [ 132.911820] Lustre: lustre-MDT0001: new disk, initializing [ 133.000857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 133.043484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 133.048251] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 136.751882] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 148.578608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 148.667061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 148.674624] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 148.879250] Lustre: lustre-OST0000: new disk, initializing [ 148.881657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 148.942847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 152.389448] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 155.169711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 155.182330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 163.369215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 163.446899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 163.453705] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 163.556185] Lustre: lustre-OST0001: new disk, initializing [ 163.558911] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 163.616836] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.773576] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 173.613692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 173.633650] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 176.858914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 182.179828] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 189.943529] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing check_logdir /tmp/testlogs/ [ 192.945159] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing yml_node [ 195.682914] Lustre: DEBUG MARKER: Client: 2.15.8.13 [ 197.307706] Lustre: DEBUG MARKER: MDS: 2.15.8.13 [ 198.917156] Lustre: DEBUG MARKER: OSS: 2.15.8.13 [ 199.965475] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Apr 14 11:35:00 EDT 2026 [ 204.990332] Lustre: DEBUG MARKER: excepting tests: 59 [ 208.902540] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing check_config_client /mnt/lustre [ 223.052844] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.257978] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 228.627122] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 231.419211] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 11:35:32 (1776180932) [ 237.672360] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 239.226767] Lustre: Failing over lustre-MDT0000 [ 239.375913] Lustre: server umount lustre-MDT0000 complete [ 239.427811] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 240.098876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 240.100160] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 240.113602] Lustre: Skipped 3 previous similar messages [ 240.121031] LustreError: Skipped 2 previous similar messages [ 244.513287] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 246.610799] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 246.622315] LustreError: Skipped 4 previous similar messages [ 247.264071] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180941/real 1776180941] req@000000004e1e0a9e x1862460512447424/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180948 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 247.293508] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 251.697971] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 251.724926] LustreError: Skipped 4 previous similar messages [ 262.233111] hrtimer: interrupt took 7102099 ns [ 262.944924] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.958487] LustreError: Skipped 15 previous similar messages [ 263.117150] LDISKFS-fs (dm-0): recovery complete [ 263.121800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 264.684943] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b378dd to 0xca62691459b37f90 [ 264.702651] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 265.079315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 266.491669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 270.315907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 270.398781] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 270.629624] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 281.041059] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 282.911019] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 290.645878] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 11:36:31 (1776180991) [ 292.551436] Lustre: Failing over lustre-OST0000 [ 292.583284] Lustre: server umount lustre-OST0000 complete [ 292.668199] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 292.683468] LustreError: Skipped 9 previous similar messages [ 293.344684] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 293.350833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.905887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.694801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 311.859350] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 312.978696] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 313.143899] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.204.134@tcp (at 0@lo) [ 313.144391] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 313.156028] Lustre: Skipped 3 previous similar messages [ 313.168738] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 315.359426] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 322.370676] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 323.405186] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 330.849877] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 11:37:11 (1776181031) [ 337.418944] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 339.242097] Lustre: Failing over lustre-MDT0000 [ 339.402048] Lustre: server umount lustre-MDT0000 complete [ 341.988567] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 341.997662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.027574] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.039224] LustreError: Skipped 12 previous similar messages [ 349.151812] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181043/real 1776181043] req@00000000c97e8ff8 x1862460512478528/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181050 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 349.179507] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.616172] LDISKFS-fs (dm-0): recovery complete [ 362.622911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 366.582554] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b37f90 to 0xca62691459b389ca [ 366.601619] Lustre: MGC192.168.204.134@tcp: Connection restored to 192.168.204.134@tcp (at 0@lo) [ 366.614792] Lustre: Skipped 1 previous similar message [ 366.867725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.799482] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 372.193252] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 372.197383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 373.602732] Lustre: lustre-MDT0000: Denying connection for new client 194251c7-7bb6-4ed1-bce6-c27bb0d8e747 (at 192.168.204.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 378.653059] Lustre: lustre-MDT0000: Denying connection for new client 194251c7-7bb6-4ed1-bce6-c27bb0d8e747 (at 192.168.204.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 383.780279] Lustre: lustre-MDT0000: Denying connection for new client 194251c7-7bb6-4ed1-bce6-c27bb0d8e747 (at 192.168.204.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 388.892697] Lustre: lustre-MDT0000: Denying connection for new client 194251c7-7bb6-4ed1-bce6-c27bb0d8e747 (at 192.168.204.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 394.027436] Lustre: lustre-MDT0000: Denying connection for new client 194251c7-7bb6-4ed1-bce6-c27bb0d8e747 (at 192.168.204.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:43 [ 404.254248] Lustre: lustre-MDT0000: Denying connection for new client 194251c7-7bb6-4ed1-bce6-c27bb0d8e747 (at 192.168.204.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:33 [ 404.273885] Lustre: Skipped 1 previous similar message [ 424.732869] Lustre: lustre-MDT0000: Denying connection for new client 194251c7-7bb6-4ed1-bce6-c27bb0d8e747 (at 192.168.204.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:13 [ 424.758994] Lustre: Skipped 3 previous similar messages [ 438.002696] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 438.011630] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 438.043644] Lustre: lustre-MDT0000: Recovery over after 1:06, of 2 clients 1 recovered and 1 was evicted. [ 438.049630] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 438.063674] Lustre: Skipped 2 previous similar messages [ 438.089964] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:97 [ 438.091321] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:65 [ 448.110742] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 11:39:08 (1776181148) [ 454.614543] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 456.542565] Lustre: Failing over lustre-MDT0000 [ 456.730653] Lustre: server umount lustre-MDT0000 complete [ 458.731249] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 458.736681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.751465] Lustre: Skipped 3 previous similar messages [ 458.759713] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 458.772370] LustreError: Skipped 32 previous similar messages [ 466.911777] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181160/real 1776181160] req@0000000077272b4f x1862460512506752/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181167 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 466.956403] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 482.090045] LDISKFS-fs (dm-0): recovery complete [ 482.093809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.301494] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b389ca to 0xca62691459b38dcf [ 483.314757] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 483.545942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.500841] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 488.940028] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 490.778723] Lustre: lustre-MDT0000: Denying connection for new client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 490.804753] Lustre: Skipped 2 previous similar messages [ 554.000181] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 554.006246] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 554.025678] Lustre: lustre-MDT0000: Recovery over after 1:06, of 2 clients 1 recovered and 1 was evicted. [ 554.031807] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 554.045478] Lustre: Skipped 3 previous similar messages [ 554.081381] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:129 [ 554.081990] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:97 [ 565.249491] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 11:41:06 (1776181266) [ 572.904207] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 574.659085] Lustre: Failing over lustre-MDT0000 [ 574.822356] Lustre: server umount lustre-MDT0000 complete [ 575.968820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.985940] Lustre: Skipped 6 previous similar messages [ 582.113471] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181276/real 1776181276] req@000000001cd608b2 x1862460512533824/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181283 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 582.139294] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 587.235869] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.247582] LustreError: Skipped 45 previous similar messages [ 596.451396] LDISKFS-fs (dm-0): recovery complete [ 596.458502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.526188] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b38dcf to 0xca62691459b3924b [ 599.790740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.357381] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 604.227929] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 605.232142] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 605.277641] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:161 [ 605.279153] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:129 [ 612.464956] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 613.817861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 620.965169] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 11:42:01 (1776181321) [ 626.942077] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 628.492208] Lustre: Failing over lustre-MDT0000 [ 628.595470] Lustre: server umount lustre-MDT0000 complete [ 630.751478] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 630.753603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.769459] Lustre: Skipped 1 previous similar message [ 637.919180] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181331/real 1776181331] req@00000000f546b1d8 x1862460512550272/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181338 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 637.940782] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 651.218537] LDISKFS-fs (dm-0): recovery complete [ 651.228753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.333486] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b3924b to 0xca62691459b3974c [ 655.352722] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 655.362451] Lustre: Skipped 5 previous similar messages [ 655.648811] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 656.227714] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 660.043653] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 661.104392] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 661.150280] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 661.157162] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:161 [ 668.207920] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 670.337369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 677.957959] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 11:42:58 (1776181378) [ 684.272885] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 686.177556] Lustre: Failing over lustre-MDT0000 [ 686.375686] Lustre: server umount lustre-MDT0000 complete [ 686.560468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.570505] Lustre: Skipped 3 previous similar messages [ 698.848102] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181392/real 1776181392] req@000000008dd77049 x1862460512567616/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181399 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 698.871493] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 707.838712] LDISKFS-fs (dm-0): recovery complete [ 707.846212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.242455] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b3974c to 0xca62691459b39ca1 [ 715.465545] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 716.063768] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 719.849930] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 720.962785] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 721.014676] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:225 [ 721.015485] Lustre: lustre-OST0001: deleting orphan objects from 0x0:163 to 0x0:193 [ 731.415463] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 734.003957] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 742.509152] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 11:44:03 (1776181443) [ 749.923698] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 751.702798] Lustre: Failing over lustre-MDT0000 [ 751.820488] Lustre: server umount lustre-MDT0000 complete [ 756.708974] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.710320] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 756.715971] Lustre: Skipped 3 previous similar messages [ 763.871371] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181457/real 1776181457] req@000000001873b5e1 x1862460512585536/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181464 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 763.908731] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 774.258677] LDISKFS-fs (dm-0): recovery complete [ 774.262447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.259833] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b39ca1 to 0xca62691459b3a1c5 [ 780.447747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 780.460564] Lustre: Skipped 2 previous similar messages [ 780.497102] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 781.598229] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 784.482136] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 785.900571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 785.908705] Lustre: Skipped 10 previous similar messages [ 786.026049] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 786.066456] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:225 [ 786.067790] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 793.796574] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 795.353636] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 802.295880] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 11:45:02 (1776181502) [ 808.632624] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 818.143684] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181512/real 1776181512] req@00000000dd3ce8cd x1862460512601984/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181519 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 818.168199] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 831.152596] LDISKFS-fs (dm-0): recovery complete [ 831.155728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.555682] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b3a1c5 to 0xca62691459b3a6db [ 835.809095] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 838.866078] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 841.308034] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:257 [ 841.309541] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:289 [ 847.584540] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 849.184955] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 856.882458] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 11:45:57 (1776181557) [ 857.876764] Lustre: *** cfs_fail_loc=13b, val=315*** [ 857.878956] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 857.894052] LustreError: 6264:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000d8bb36e0 x1862460497793664/t38654705666(0) o35->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:274/0 lens 392/456 e 0 to 0 dl 1776181564 ref 1 fl Interpret:/0/0 rc 0/0 job:'openfile.0' [ 865.085690] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 865.100256] Lustre: 6264:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000eadca684 x1862460497793664/t38654705666(0) o35->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:282/0 lens 392/456 e 0 to 0 dl 1776181572 ref 1 fl Interpret:/2/0 rc 0/0 job:'openfile.0' [ 865.349653] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 866.811882] Lustre: Failing over lustre-MDT0000 [ 866.816684] Lustre: Skipped 1 previous similar message [ 866.934359] Lustre: server umount lustre-MDT0000 complete [ 866.937441] Lustre: Skipped 1 previous similar message [ 870.184281] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.205455] LustreError: Skipped 177 previous similar messages [ 871.904436] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 889.059810] LDISKFS-fs (dm-0): recovery complete [ 889.062748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.734197] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 897.427941] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 897.435350] Lustre: Skipped 1 previous similar message [ 902.246905] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 902.252459] Lustre: Skipped 1 previous similar message [ 902.285849] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 902.305240] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:321 [ 902.314633] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:289 [ 915.699405] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 917.514681] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 927.759659] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 11:47:07 (1776181627) [ 935.376754] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 937.962783] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.975357] Lustre: Skipped 13 previous similar messages [ 950.241311] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181644/real 1776181644] req@0000000064e03c0c x1862460512640192/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181651 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 950.259094] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 950.264895] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 950.271844] LustreError: Skipped 1 previous similar message [ 957.572845] LDISKFS-fs (dm-0): recovery complete [ 957.579068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.650980] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b3abd5 to 0xca62691459b3b090 [ 967.658016] Lustre: Skipped 1 previous similar message [ 967.933184] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 971.415654] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 973.393067] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:321 [ 973.405043] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:353 [ 980.342339] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 981.744547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 988.317790] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 11:48:09 (1776181689) [ 994.901056] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 995.646148] Lustre: *** cfs_fail_loc=114, val=0*** [ 997.700304] Lustre: Failing over lustre-MDT0000 [ 997.706913] Lustre: Skipped 1 previous similar message [ 997.847152] Lustre: server umount lustre-MDT0000 complete [ 997.851638] Lustre: Skipped 1 previous similar message [ 1017.751907] LDISKFS-fs (dm-0): recovery complete [ 1017.757461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.580442] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1025.595510] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1027.648599] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:353 [ 1027.652048] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:385 [ 1033.570605] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1035.066586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1042.502771] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 11:49:03 (1776181743) [ 1047.792697] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1048.539077] Lustre: *** cfs_fail_loc=128, val=0*** [ 1053.163371] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1072.008535] LDISKFS-fs (dm-0): recovery complete [ 1072.011894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1077.225778] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 1077.231459] Lustre: Skipped 23 previous similar messages [ 1077.478792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1077.487440] Lustre: Skipped 4 previous similar messages [ 1077.518689] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1078.057612] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1078.076835] Lustre: Skipped 2 previous similar messages [ 1081.682735] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1082.931853] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1082.942809] Lustre: Skipped 2 previous similar messages [ 1082.978155] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:385 [ 1082.986474] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:417 [ 1090.996063] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1092.709856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1100.431345] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 11:50:00 (1776181800) [ 1107.310621] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1113.569191] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1120.607145] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181814/real 1776181814] req@00000000e1916cc0 x1862460512690624/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181821 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 1120.629709] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1120.638103] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1120.649034] LustreError: Skipped 2 previous similar messages [ 1130.188647] LDISKFS-fs (dm-0): recovery complete [ 1130.193348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1137.126731] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b3b9ff to 0xca62691459b3c034 [ 1137.134015] Lustre: Skipped 2 previous similar messages [ 1141.178478] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1143.028681] Lustre: lustre-OST0001: deleting orphan objects from 0x0:391 to 0x0:417 [ 1143.028917] Lustre: lustre-OST0000: deleting orphan objects from 0x0:423 to 0x0:449 [ 1150.012298] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1151.995071] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1162.320285] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 11:51:02 (1776181862) [ 1171.119205] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1193.920522] LDISKFS-fs (dm-0): recovery complete [ 1193.923383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.375982] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1198.382808] Lustre: Skipped 1 previous similar message [ 1202.135532] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1203.871344] Lustre: lustre-OST0000: deleting orphan objects from 0x0:455 to 0x0:481 [ 1203.872372] Lustre: lustre-OST0001: deleting orphan objects from 0x0:423 to 0x0:449 [ 1211.341873] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1212.842386] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1219.890893] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 11:52:00 (1776181920) [ 1227.905170] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1239.523344] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.548796] Lustre: Skipped 18 previous similar messages [ 1257.053425] LDISKFS-fs (dm-0): recovery complete [ 1257.057884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.543769] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1272.616151] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:609 [ 1272.619466] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:577 [ 1277.456757] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1278.974491] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1298.530220] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 11:53:19 (1776181999) [ 1306.044740] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1308.192940] Lustre: Failing over lustre-MDT0000 [ 1308.194724] Lustre: Skipped 4 previous similar messages [ 1308.347935] Lustre: server umount lustre-MDT0000 complete [ 1308.350694] Lustre: Skipped 4 previous similar messages [ 1308.639787] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1331.535972] LDISKFS-fs (dm-0): recovery complete [ 1331.541702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.411060] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1333.437830] Lustre: Skipped 1 previous similar message [ 1338.973380] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:641 [ 1338.990642] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:609 [ 1339.185885] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1349.339074] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1350.724158] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1359.725085] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 11:54:20 (1776182060) [ 1365.453433] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1376.735139] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182070/real 1776182070] req@00000000d613179f x1862460512801216/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182077 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1376.766650] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1376.772280] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1376.788057] LustreError: Skipped 3 previous similar messages [ 1382.885775] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1382.915348] LustreError: Skipped 318 previous similar messages [ 1389.163060] LDISKFS-fs (dm-0): recovery complete [ 1389.166626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1393.441818] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1393.447915] Lustre: Skipped 4 previous similar messages [ 1397.932690] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1398.867756] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1398.873208] Lustre: Skipped 4 previous similar messages [ 1398.927788] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:641 [ 1398.927804] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:673 [ 1406.591346] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1408.341420] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1416.491940] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 11:55:16 (1776182116) [ 1425.176543] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1427.111252] LustreError: 49296:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000fa84f5a0) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1448.445919] LDISKFS-fs (dm-0): recovery complete [ 1448.449771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1454.055470] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b45dd7 to 0xca62691459b462d8 [ 1454.066050] Lustre: Skipped 4 previous similar messages [ 1457.997841] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1459.770119] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:673 [ 1459.771270] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:705 [ 1465.511465] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1466.912564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1472.485663] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 11:56:13 (1776182173) [ 1477.263798] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1480.161461] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1480.173891] LustreError: Skipped 1 previous similar message [ 1498.543417] LDISKFS-fs (dm-0): recovery complete [ 1498.547293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.596907] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:705 [ 1509.601523] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:737 [ 1509.905051] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1521.620354] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1523.663751] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1530.988110] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 11:57:11 (1776182231) [ 1538.307856] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1563.447577] LDISKFS-fs (dm-0): recovery complete [ 1563.450599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.877418] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1568.815249] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:769 [ 1568.815461] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:737 [ 1574.004610] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1575.376634] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1581.848648] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 11:58:02 (1776182282) [ 1587.314456] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1610.388972] LDISKFS-fs (dm-0): recovery complete [ 1610.392456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.809102] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 1613.814474] Lustre: Skipped 44 previous similar messages [ 1614.037575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1614.039742] Lustre: Skipped 8 previous similar messages [ 1614.070889] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1614.079126] Lustre: Skipped 4 previous similar messages [ 1617.546792] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1619.537945] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:769 [ 1619.538712] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:801 [ 1626.442614] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1628.066511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1635.604843] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 11:58:56 (1776182336) [ 1641.877960] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1665.894116] LDISKFS-fs (dm-0): recovery complete [ 1665.897294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.135486] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1673.858320] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:801 [ 1673.859124] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:833 [ 1680.073692] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1681.474765] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1687.053554] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 11:59:48 (1776182388) [ 1691.990721] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1711.472107] LDISKFS-fs (dm-0): recovery complete [ 1711.474748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.325875] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1723.993494] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:865 [ 1723.993967] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:833 [ 1731.325599] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1732.647788] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1739.498449] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 12:00:40 (1776182440) [ 1746.858961] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1749.475550] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1749.481365] LustreError: Skipped 4 previous similar messages [ 1774.927520] LDISKFS-fs (dm-0): recovery complete [ 1774.934120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1783.844095] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1783.867451] Lustre: lustre-OST0001: deleting orphan objects from 0x0:835 to 0x0:865 [ 1783.873482] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:897 [ 1794.292213] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1795.409666] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1801.811362] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 12:01:42 (1776182502) [ 1809.142349] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1814.496038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1814.509845] Lustre: Skipped 40 previous similar messages [ 1832.668969] LDISKFS-fs (dm-0): recovery complete [ 1832.672972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.730222] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1844.790639] Lustre: lustre-OST0000: deleting orphan objects from 0x0:899 to 0x0:929 [ 1844.792934] Lustre: lustre-OST0001: deleting orphan objects from 0x0:835 to 0x0:897 [ 1850.071496] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1851.816396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1858.467610] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 12:02:39 (1776182559) [ 1864.910259] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1867.004268] Lustre: Failing over lustre-MDT0000 [ 1867.008186] Lustre: Skipped 9 previous similar messages [ 1867.154725] Lustre: server umount lustre-MDT0000 complete [ 1867.157099] Lustre: Skipped 9 previous similar messages [ 1895.517826] LDISKFS-fs (dm-0): recovery complete [ 1895.528271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.073801] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1911.082092] Lustre: Skipped 8 previous similar messages [ 1914.802607] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1916.024726] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1916.031278] Lustre: Skipped 8 previous similar messages [ 1916.076412] Lustre: lustre-OST0001: deleting orphan objects from 0x0:899 to 0x0:929 [ 1923.128736] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1924.388287] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1931.680904] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 12:03:52 (1776182632) [ 1937.805893] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1948.639379] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182642/real 1776182642] req@00000000468605ae x1862460512971264/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182649 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1948.654249] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 1948.661541] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1948.668562] LustreError: Skipped 9 previous similar messages [ 1961.840937] LDISKFS-fs (dm-0): recovery complete [ 1961.843974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.203709] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1971.841430] Lustre: lustre-OST0001: deleting orphan objects from 0x0:931 to 0x0:961 [ 1971.850422] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:961 [ 1982.179806] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1983.863924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1992.345893] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 12:04:52 (1776182692) [ 2001.445906] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2007.334061] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.361556] LustreError: Skipped 389 previous similar messages [ 2025.892523] LDISKFS-fs (dm-0): recovery complete [ 2025.895641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.591564] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b492d5 to 0xca62691459b497f9 [ 2031.598038] Lustre: Skipped 9 previous similar messages [ 2036.314556] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2037.308047] Lustre: lustre-OST0001: deleting orphan objects from 0x0:963 to 0x0:993 [ 2037.308735] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:993 [ 2045.632927] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2047.472469] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2056.409428] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 12:05:56 (1776182756) [ 2063.204832] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2091.169943] LDISKFS-fs (dm-0): recovery complete [ 2091.174653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.003429] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2097.837430] Lustre: lustre-OST0000: deleting orphan objects from 0x0:995 to 0x0:1025 [ 2097.839492] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1025 [ 2108.319927] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2110.138866] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2119.303729] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 12:06:59 (1776182819) [ 2126.655650] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2128.879573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2151.800955] LDISKFS-fs (dm-0): recovery complete [ 2151.804604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.836537] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2158.856728] Lustre: Skipped 8 previous similar messages [ 2164.468619] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2164.475245] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2164.590864] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2175.555227] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2177.596534] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2186.167108] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:08:06 (1776182886) [ 2195.062449] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2222.597845] LDISKFS-fs (dm-0): recovery complete [ 2222.605211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.597343] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 2223.600042] Lustre: Skipped 49 previous similar messages [ 2223.768170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.776384] Lustre: Skipped 9 previous similar messages [ 2228.551722] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2229.334266] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1059 to 0x0:1089 [ 2229.334458] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1089 [ 2238.148105] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2240.339337] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2249.227894] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 12:09:09 (1776182949) [ 2253.273266] Lustre: 77535:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e5918ffb-1455-4fdd-ac94-c09c5379639f at adminstrative request [ 2265.055842] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2265.071396] LustreError: Skipped 4 previous similar messages [ 2280.494753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.186914] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2294.301335] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1121 [ 2294.301488] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1091 to 0x0:1121 [ 2302.892913] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2304.079612] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2308.238142] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2313.522268] Lustre: DEBUG MARKER: before 2844, after 2844 [ 2318.654599] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 12:10:19 (1776183019) [ 2320.363826] Lustre: 79570:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e5918ffb-1455-4fdd-ac94-c09c5379639f at adminstrative request [ 2334.438890] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 12:10:33 (1776183033) [ 2342.723272] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2368.631185] LDISKFS-fs (dm-0): recovery complete [ 2368.635030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.831214] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2375.866970] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1124 to 0x0:1153 [ 2375.866969] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 2384.887968] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2386.526751] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2393.549673] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:11:34 (1776183094) [ 2400.480323] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2425.700354] LDISKFS-fs (dm-0): recovery complete [ 2425.704775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.696145] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 2436.698172] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 2436.962702] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2446.362419] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2447.609659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2454.721601] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 12:12:35 (1776183155) [ 2461.307732] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2467.296358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2467.309045] Lustre: Skipped 36 previous similar messages [ 2487.335065] LDISKFS-fs (dm-0): recovery complete [ 2487.338513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.223429] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2497.112409] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2497.116281] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2505.516412] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2507.042067] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2514.656075] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 12:13:35 (1776183215) [ 2520.803761] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2522.433564] Lustre: Failing over lustre-MDT0000 [ 2522.437911] Lustre: Skipped 9 previous similar messages [ 2522.569161] Lustre: server umount lustre-MDT0000 complete [ 2522.571524] Lustre: Skipped 9 previous similar messages [ 2541.832880] LDISKFS-fs (dm-0): recovery complete [ 2541.835738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.640658] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2553.655330] Lustre: Skipped 9 previous similar messages [ 2558.474325] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2558.491329] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2558.514709] Lustre: Skipped 9 previous similar messages [ 2558.564339] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2558.570568] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2568.115068] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2569.730969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2576.719872] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:14:37 (1776183277) [ 2582.246299] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2590.689826] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183284/real 1776183284] req@00000000752e1749 x1862460513162304/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183291 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 2590.717577] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 2590.722331] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2590.730095] LustreError: Skipped 9 previous similar messages [ 2607.688525] LDISKFS-fs (dm-0): recovery complete [ 2607.691809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2607.905789] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.918611] LustreError: Skipped 387 previous similar messages [ 2627.195286] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2629.158096] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2629.160883] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1219 to 0x0:1281 [ 2636.064102] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2637.750501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2644.800991] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:15:45 (1776183345) [ 2650.391649] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2672.647427] LDISKFS-fs (dm-0): recovery complete [ 2672.651829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.248658] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b4c98c to 0xca62691459b4ceef [ 2678.258504] Lustre: Skipped 9 previous similar messages [ 2682.494761] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2684.019771] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 2684.020336] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 2690.438245] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2691.665070] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2698.055734] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:16:39 (1776183399) [ 2705.020952] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2729.481758] LDISKFS-fs (dm-0): recovery complete [ 2729.484613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.955335] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2739.894853] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2739.897468] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2750.225236] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2752.269742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2760.559872] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:17:41 (1776183461) [ 2767.810596] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2793.303643] LDISKFS-fs (dm-0): recovery complete [ 2793.318801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.811286] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2793.818967] Lustre: Skipped 9 previous similar messages [ 2798.481725] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2799.213659] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2799.219527] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2808.510709] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2810.088524] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2817.528851] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:18:38 (1776183518) [ 2825.475924] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2851.816193] LDISKFS-fs (dm-0): recovery complete [ 2851.820287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.885447] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 2854.897217] Lustre: Skipped 49 previous similar messages [ 2855.195308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2855.201447] Lustre: Skipped 9 previous similar messages [ 2860.234338] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2860.619893] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 2860.620376] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 2869.488808] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2870.902854] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2879.013590] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:19:39 (1776183579) [ 2885.925112] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2891.234500] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2891.238837] LustreError: Skipped 6 previous similar messages [ 2908.701886] LDISKFS-fs (dm-0): recovery complete [ 2908.707425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.712029] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2921.562860] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 2921.564844] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 2930.007857] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2931.652603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2938.684851] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 12:20:39 (1776183639) [ 2945.088808] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2968.140456] LDISKFS-fs (dm-0): recovery complete [ 2968.147780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2968.354818] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (not set up) [ 2973.065547] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2973.744354] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 2973.744387] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 2981.002391] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2982.456565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2989.520476] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 12:21:30 (1776183690) [ 2990.475535] Lustre: 102266:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e5918ffb-1455-4fdd-ac94-c09c5379639f at adminstrative request [ 3002.265898] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 12:21:42 (1776183702) [ 3008.447554] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3022.871591] LDISKFS-fs (dm-0): recovery complete [ 3022.876587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3028.155505] LustreError: 103548:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3028.161977] LustreError: 103548:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3028.167401] Lustre: 103582:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3028.183541] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3028.197136] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3028.209084] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3028.255613] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1475 to 0x0:1505 [ 3028.289339] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1443 to 0x0:1505 [ 3032.171808] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3033.577215] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3050.590109] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 12:22:30 (1776183750) [ 3056.841938] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3069.647492] LDISKFS-fs (dm-0): recovery complete [ 3069.650969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.681677] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3072.700588] LustreError: 105465:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3072.706402] LustreError: 105465:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3072.712336] Lustre: 105498:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3072.718612] Lustre: 105498:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3072.723470] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3072.730980] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3072.745085] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3072.780940] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1511 to 0x0:1537 [ 3072.781716] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1511 to 0x0:1537 [ 3076.507221] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3078.120567] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3089.322692] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3089.326963] Lustre: Skipped 1 previous similar message [ 3095.801806] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 12:23:16 (1776183796) [ 3102.302299] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3103.723773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.725871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.731852] Lustre: Skipped 46 previous similar messages [ 3103.746016] Lustre: Skipped 3 previous similar messages [ 3113.081551] LDISKFS-fs (dm-0): recovery complete [ 3113.089563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3113.353187] LustreError: 107384:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3113.359359] LustreError: 107384:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3113.364849] Lustre: 107417:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3113.370589] Lustre: 107417:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3113.377472] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3113.387520] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3113.405365] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3113.455957] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1543 to 0x0:1569 [ 3113.456258] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1544 to 0x0:1569 [ 3116.616270] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3118.570606] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3134.073704] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 12:23:54 (1776183834) [ 3134.986663] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3134.989959] LustreError: 9706:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000005097dfba x1862460498328512/t201863462916(0) o36->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:286/0 lens 504/456 e 0 to 0 dl 1776183841 ref 1 fl Interpret:/0/0 rc 0/0 job:'rm.0' [ 3138.499644] Lustre: Failing over lustre-MDT0000 [ 3138.504921] Lustre: Skipped 10 previous similar messages [ 3138.695749] Lustre: server umount lustre-MDT0000 complete [ 3138.698125] Lustre: Skipped 10 previous similar messages [ 3145.962803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3152.154590] LustreError: 108763:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3152.160614] LustreError: 108763:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3152.167077] Lustre: 108796:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3152.172875] Lustre: 108796:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3152.178630] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3152.185341] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3152.199840] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3152.251483] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1571 to 0x0:1601 [ 3152.255158] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1544 to 0x0:1601 [ 3156.600160] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3157.474840] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3177.895636] Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ============ 12:24:38 (1776183878) [ 3186.469394] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3200.463183] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183894/real 1776183894] req@00000000c7604306 x1862460513357056/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183901 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 3200.485167] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 3200.491696] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3200.513231] LustreError: Skipped 10 previous similar messages [ 3211.037761] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.049361] LustreError: Skipped 336 previous similar messages [ 3211.650077] LDISKFS-fs (dm-0): recovery complete [ 3211.652779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.405793] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3219.417875] Lustre: Skipped 7 previous similar messages [ 3222.945709] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3223.580472] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3223.585716] Lustre: Skipped 7 previous similar messages [ 3223.617921] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1571 to 0x0:1633 [ 3223.618149] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1603 to 0x0:1633 [ 3231.930677] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 12:25:32 (1776183932) [ 3239.310425] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3251.859525] LDISKFS-fs (dm-0): recovery complete [ 3251.863859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.587385] LustreError: 112527:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3256.596393] LustreError: 112527:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3256.603068] Lustre: 112561:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3256.608253] Lustre: 112561:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3256.614280] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3256.621450] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3256.637549] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3256.684792] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1603 to 0x0:1665 [ 3256.695301] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1571 to 0x0:1665 [ 3260.473593] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3261.944053] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3280.459651] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:26:21 (1776183981) [ 3307.857308] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3336.046486] LDISKFS-fs (dm-0): recovery complete [ 3336.055974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.193888] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b51202 to 0xca62691459b5ba5d [ 3337.214096] Lustre: Skipped 11 previous similar messages [ 3342.952697] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3342.989421] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2066 to 0x0:2081 [ 3342.992421] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2066 to 0x0:2081 [ 3352.730989] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3354.963697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3374.724199] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:27:55 (1776184075) [ 3396.430601] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3422.475401] LDISKFS-fs (dm-0): recovery complete [ 3422.484594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.047574] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3429.053630] Lustre: Skipped 20 previous similar messages [ 3432.268822] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3437.913234] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2482 to 0x0:2497 [ 3437.919908] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2482 to 0x0:2497 [ 3442.914344] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3444.279973] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3462.617986] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 12:29:23 (1776184163) [ 3463.838427] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 3465.282619] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 12:29:26 (1776184166) [ 3467.035286] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3467.886965] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3467.902924] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3467.916940] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to (at 0@lo) [ 3467.920775] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2482 to 0x0:2529 [ 3467.923304] Lustre: Skipped 54 previous similar messages [ 3473.167950] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 12:29:34 (1776184174) [ 3493.646816] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3504.608097] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.616322] LustreError: Skipped 9 previous similar messages [ 3524.982647] LDISKFS-fs (dm-2): recovery complete [ 3524.987354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3525.219278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.226075] Lustre: Skipped 10 previous similar messages [ 3527.393267] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:2945 [ 3527.413988] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3528.744207] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3582.954447] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 12:31:23 (1776184283) [ 3590.238680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3614.569549] LDISKFS-fs (dm-0): recovery complete [ 3614.576736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.454803] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3627.037371] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3627.038037] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:2977 [ 3634.936529] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3636.227538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3653.609701] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 12:32:34 (1776184354) [ 3657.862830] LustreError: 20543:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3663.327131] LustreError: 20543:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3663.334181] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 3663.392597] LustreError: 44514:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3665.250486] LustreError: 9706:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3670.495328] LustreError: 9706:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3670.503141] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 3671.007849] LustreError: 122370:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3671.017291] LustreError: 8234:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3671.032923] LustreError: 8234:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 3672.033955] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:2945 [ 3672.385838] LustreError: 6261:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3677.663357] LustreError: 6261:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3677.675818] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 3677.685832] Lustre: Skipped 1 previous similar message [ 3679.336614] LustreError: 7728:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3684.831128] LustreError: 7728:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3684.839596] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 3686.555334] LustreError: 6263:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3691.999318] LustreError: 6263:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3698.655221] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 3698.660742] Lustre: Skipped 1 previous similar message [ 3700.253363] LustreError: 20543:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3700.257307] LustreError: 20543:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message [ 3705.311227] LustreError: 20543:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3705.321206] LustreError: 20543:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message [ 3718.623484] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 3718.630523] Lustre: Skipped 2 previous similar messages [ 3720.142892] LustreError: 7728:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3720.152592] LustreError: 7728:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages [ 3725.279245] LustreError: 7728:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3725.288444] LustreError: 7728:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages [ 3733.189162] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 12:33:53 (1776184433) [ 3734.815759] LustreError: 9706:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3740.968355] Lustre: lustre-MDT0000: Export 00000000d8ce3541 already connecting from 192.168.204.34@tcp [ 3742.766560] Lustre: lustre-MDT0000: Export 00000000d8ce3541 already connecting from 192.168.204.34@tcp [ 3744.465039] Lustre: lustre-MDT0000: Export 00000000d8ce3541 already connecting from 192.168.204.34@tcp [ 3747.901083] Lustre: lustre-MDT0000: Export 00000000d8ce3541 already connecting from 192.168.204.34@tcp [ 3747.909849] Lustre: Skipped 2 previous similar messages [ 3753.361449] Lustre: lustre-MDT0000: Export 00000000d8ce3541 already connecting from 192.168.204.34@tcp [ 3753.370630] Lustre: Skipped 3 previous similar messages [ 3759.449860] LustreError: 9706:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 3759.459786] Lustre: 9706:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/5s); client may timeout req@00000000dad6079e x1862460499359680/t0(0) o38->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:0/0 lens 520/416 e 0 to 0 dl 1776184455 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0' [ 3760.413740] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 3760.435548] Lustre: Skipped 2 previous similar messages [ 3764.638701] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 12:34:24 (1776184464) [ 3772.095426] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3775.366625] Lustre: Failing over lustre-MDT0000 [ 3775.370512] Lustre: Skipped 6 previous similar messages [ 3775.455745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3775.489820] Lustre: Skipped 28 previous similar messages [ 3775.492114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3775.574875] Lustre: server umount lustre-MDT0000 complete [ 3775.579648] Lustre: Skipped 6 previous similar messages [ 3787.794592] LDISKFS-fs (dm-0): recovery complete [ 3787.798596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.955249] Lustre: *** cfs_fail_loc=712, val=0*** [ 3791.959145] LustreError: 44523:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery req@0000000081875acf x1862460513716224/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' [ 3791.982337] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3792.065400] LustreError: 127051:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3792.072961] LustreError: 127051:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3792.082842] Lustre: 127086:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3792.090276] Lustre: 127086:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3792.100391] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3792.110971] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0] [ 3792.131602] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3792.189022] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:3009 [ 3792.193549] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:2977 [ 3796.549966] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3797.476236] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3812.840195] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3812.853126] LustreError: Skipped 192 previous similar messages [ 3819.986034] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184513/real 1776184513] req@0000000024a26f7b x1862460513725696/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184520 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 3820.007366] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 3820.021748] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3820.032787] LustreError: Skipped 5 previous similar messages [ 3829.982448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.221422] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3837.225758] Lustre: Skipped 4 previous similar messages [ 3840.585265] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3842.048698] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3842.055764] Lustre: Skipped 4 previous similar messages [ 3842.078324] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2930 to 0x0:3009 [ 3842.078656] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2899 to 0x0:3041 [ 3850.198775] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3851.598349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3859.227204] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:35:59 (1776184559) [ 3859.405220] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 3866.778761] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:36:07 (1776184567) [ 3867.813172] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3867.816450] LustreError: 6271:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000fbac65f1 x1862460499385664/t0(0) o700->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:264/0 lens 264/248 e 0 to 0 dl 1776184574 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0' [ 3877.858322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3895.631930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3910.861948] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3912.216157] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3043 to 0x0:3073 [ 3912.217932] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3011 to 0x0:3041 [ 3919.698731] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3921.156573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3929.411157] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:37:10 (1776184630) [ 3932.129835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3932.144447] Lustre: Skipped 3 previous similar messages [ 3951.993487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3953.879230] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3084 to 0x0:3105 [ 3953.883149] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97 [ 3955.257431] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3963.461887] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3964.864663] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4035.566268] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:38:56 (1776184736) [ 4043.056827] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4069.969852] LDISKFS-fs (dm-0): recovery complete [ 4069.974083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.449219] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b84303 to 0xca62691459b85387 [ 4073.467901] Lustre: Skipped 5 previous similar messages [ 4073.476411] Lustre: MGC192.168.204.134@tcp: Connection restored to 192.168.204.134@tcp (at 0@lo) [ 4073.486313] Lustre: Skipped 26 previous similar messages [ 4073.727670] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4073.736746] Lustre: Skipped 8 previous similar messages [ 4078.681744] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4079.655313] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3062 to 0x0:3105 [ 4079.656416] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137 [ 4152.215726] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:40:52 (1776184852) [ 4154.210578] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4154.217403] Lustre: Skipped 2 previous similar messages [ 4154.226449] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169 [ 4155.245580] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3201 [ 4167.228260] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:41:07 (1776184867) [ 4191.787506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.166126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.178347] Lustre: Skipped 6 previous similar messages [ 4201.475383] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4201.496466] LustreError: 135198:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000b1fa22d9 x1862460499441088/t0(0) o101->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:623/0 lens 328/344 e 0 to 0 dl 1776184933 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0' [ 4201.711573] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4234.072384] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnected, waiting for 2 clients in recovery for 0:56 [ 4234.225488] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3203 to 0x0:3233 [ 4234.233022] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3116 to 0x0:3137 [ 4241.647364] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4243.551420] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4254.771215] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:42:34 (1776184954) [ 4257.301289] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4266.489374] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4270.062286] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4270.075552] LustreError: Skipped 5 previous similar messages [ 4293.493730] LDISKFS-fs (dm-0): recovery complete [ 4293.502782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.555694] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3203 to 0x0:3265 [ 4303.567504] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3169 [ 4304.795714] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4319.638880] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4322.297375] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4332.935237] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:43:53 (1776185033) [ 4334.507406] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4344.750843] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4370.347180] LDISKFS-fs (dm-0): recovery complete [ 4370.350608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4379.452793] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4379.684433] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3297 [ 4379.687151] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3201 [ 4389.181805] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4390.815670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4400.240540] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:45:00 (1776185100) [ 4401.355122] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4410.672307] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 4410.690476] Lustre: Skipped 2 previous similar messages [ 4411.516717] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4413.692267] Lustre: Failing over lustre-MDT0000 [ 4413.701890] Lustre: Skipped 7 previous similar messages [ 4413.967808] Lustre: server umount lustre-MDT0000 complete [ 4413.972302] Lustre: Skipped 7 previous similar messages [ 4415.458260] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4415.461946] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4415.485322] Lustre: Skipped 34 previous similar messages [ 4415.514538] LustreError: Skipped 239 previous similar messages [ 4422.623146] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185116/real 1776185116] req@00000000f7d11b8f x1862460513893504/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185123 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 4422.660088] Lustre: 3352:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 4422.668597] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4422.676880] LustreError: Skipped 5 previous similar messages [ 4440.480789] LDISKFS-fs (dm-0): recovery complete [ 4440.487827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.715401] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4455.726243] Lustre: Skipped 6 previous similar messages [ 4460.564395] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4461.105979] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4461.118266] Lustre: Skipped 6 previous similar messages [ 4461.158294] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233 [ 4461.162727] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3329 [ 4471.505651] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:46:12 (1776185172) [ 4473.623145] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4473.626665] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4473.634774] LustreError: 44256:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000fc3000dd x1862460499473024/t261993005072(0) o35->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:115/0 lens 392/456 e 0 to 0 dl 1776185180 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4497.479269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.138500] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4511.792052] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3361 [ 4511.792164] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3265 [ 4511.823416] Lustre: 44256:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000ee48a4d0 x1862460499473024/t261993005072(0) o35->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:153/0 lens 392/456 e 0 to 0 dl 1776185218 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4521.239229] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4523.179971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4531.887745] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:47:12 (1776185232) [ 4533.251631] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4533.254517] LustreError: 7728:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000710df885 x1862460499481792/t266287972368(0) o36->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:204/0 lens 504/448 e 0 to 0 dl 1776185269 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4544.092704] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4573.217508] LDISKFS-fs (dm-0): recovery complete [ 4573.221113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4593.161588] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4593.705141] Lustre: 20543:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000003b7560be x1862460499481792/t266287972368(0) o36->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:264/0 lens 504/448 e 0 to 0 dl 1776185329 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4593.722748] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3363 to 0x0:3393 [ 4593.733529] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3297 [ 4602.825885] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4604.619588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4613.792531] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:48:33 (1776185313) [ 4615.349085] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4615.362521] LustreError: 6263:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000b44cffa7 x1862460499491264/t270582939664(0) o36->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:286/0 lens 504/448 e 0 to 0 dl 1776185351 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4617.271943] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4624.685629] Lustre: 9706:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000004b85bd01 x1862460499491264/t270582939664(0) o36->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:295/0 lens 504/448 e 0 to 0 dl 1776185360 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4624.709064] Lustre: 9706:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4626.529181] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4652.580787] LDISKFS-fs (dm-0): recovery complete [ 4652.584838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4662.353749] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4663.821085] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3329 [ 4663.821431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425 [ 4673.558296] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:49:34 (1776185374) [ 4674.610532] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4674.621829] Lustre: Skipped 1 previous similar message [ 4674.626664] LustreError: 6263:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000207fbf5a x1862460499499392/t274877906960(0) o36->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:345/0 lens 504/448 e 0 to 0 dl 1776185410 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4674.642285] LustreError: 6263:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 4676.421268] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4676.424814] Lustre: Skipped 1 previous similar message [ 4683.576851] Lustre: 20543:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000e7b2d174 x1862460499499392/t274877906960(0) o36->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:354/0 lens 504/448 e 0 to 0 dl 1776185419 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4683.930106] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4713.671870] LDISKFS-fs (dm-0): recovery complete [ 4713.675329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.958457] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459b88bcd to 0xca62691459b89233 [ 4713.975569] Lustre: Skipped 7 previous similar messages [ 4713.992242] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 4714.001246] Lustre: Skipped 41 previous similar messages [ 4714.307744] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4714.324451] Lustre: Skipped 7 previous similar messages [ 4719.146416] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4719.738862] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3361 [ 4719.741639] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3457 [ 4730.494799] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:50:30 (1776185430) [ 4731.868336] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4734.314632] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4734.316677] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4734.337195] LustreError: 44256:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000053d91b9d x1862460499507264/t279172874256(0) o35->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:376/0 lens 392/456 e 0 to 0 dl 1776185441 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4741.421647] Lustre: 44256:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000202f4b74 x1862460499507264/t279172874256(0) o35->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:383/0 lens 392/456 e 0 to 0 dl 1776185448 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4744.595930] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4775.530903] LDISKFS-fs (dm-0): recovery complete [ 4775.540207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.602590] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4795.979333] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3393 [ 4795.982501] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3489 [ 4808.127516] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:51:47 (1776185507) [ 4809.419642] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4809.465172] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4809.470853] LustreError: 6261:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000adf86ff2 x1862460499514048/t283467841549(0) o101->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:480/0 lens 664/600 e 0 to 0 dl 1776185545 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0' [ 4845.934974] Lustre: 6263:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000a39755d3 x1862460499514048/t283467841549(0) o101->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:516/0 lens 664/3424 e 0 to 0 dl 1776185581 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0' [ 4852.868925] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:52:33 (1776185553) [ 4859.792285] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4884.926659] LDISKFS-fs (dm-0): recovery complete [ 4884.936727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4887.276995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.281988] Lustre: Skipped 8 previous similar messages [ 4892.200650] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4892.751371] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521 [ 4892.761513] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3425 [ 4902.981507] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4904.855754] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4925.882183] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:53:46 (1776185626) [ 4934.110353] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4938.727205] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4938.739978] LustreError: Skipped 5 previous similar messages [ 4961.003364] LDISKFS-fs (dm-0): recovery complete [ 4961.007481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4967.383845] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4968.012472] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3457 [ 4968.012479] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3523 to 0x0:3553 [ 4977.464435] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4979.545525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4984.986445] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4998.780812] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:54:58 (1776185698) [ 5059.944917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5061.718133] Lustre: Failing over lustre-MDT0000 [ 5061.721490] Lustre: Skipped 7 previous similar messages [ 5062.118244] Lustre: server umount lustre-MDT0000 complete [ 5062.122137] Lustre: Skipped 7 previous similar messages [ 5065.187419] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.188333] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5065.206845] Lustre: Skipped 31 previous similar messages [ 5065.244508] LustreError: Skipped 365 previous similar messages [ 5072.351114] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185766/real 1776185766] req@0000000061eaa0bc x1862460514071296/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185773 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 5072.373807] Lustre: 3353:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 5072.393101] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5072.412148] LustreError: Skipped 7 previous similar messages [ 5088.101065] LDISKFS-fs (dm-0): recovery complete [ 5088.104965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5091.222937] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5091.235605] Lustre: Skipped 7 previous similar messages [ 5095.313860] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5095.742047] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5095.756766] Lustre: Skipped 7 previous similar messages [ 5095.817840] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4708 to 0x0:4737 [ 5095.820355] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4804 to 0x0:4833 [ 5105.348731] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5107.276221] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5192.997331] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:58:13 (1776185893) [ 5204.545984] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5234.126844] LDISKFS-fs (dm-0): recovery complete [ 5234.143902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.287280] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5254.812676] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4739 to 0x0:4769 [ 5254.814955] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4804 to 0x0:4865 [ 5263.711351] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5265.461514] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5274.983668] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5276.548300] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5283.909664] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:59:44 (1776185984) [ 5292.479779] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5328.192046] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 5328.200894] Lustre: Skipped 4 previous similar messages [ 5331.388814] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5331.392412] LustreError: 7728:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000000af55304 x1862460501005056/t300647710728(0) o36->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:247/0 lens 66040/440 e 0 to 0 dl 1776186067 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0' [ 5369.144629] Lustre: 6262:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000095f92d9 x1862460501005056/t300647710728(0) o36->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:285/0 lens 66040/440 e 0 to 0 dl 1776186105 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0' [ 5381.190767] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5383.397746] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 13:01:23 (1776186083) [ 5398.188528] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5426.092602] LDISKFS-fs (dm-0): recovery complete [ 5426.096519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5427.686583] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459bcb09a to 0xca62691459bcded0 [ 5427.711617] Lustre: Skipped 5 previous similar messages [ 5427.714806] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 5427.722800] Lustre: Skipped 29 previous similar messages [ 5428.001574] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5428.004597] Lustre: Skipped 5 previous similar messages [ 5432.641555] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5434.586416] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4871 to 0x0:4897 [ 5434.588649] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4966 to 0x0:4993 [ 5444.366739] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5447.884210] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5462.137990] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 13:02:42 (1776186162) [ 5485.239832] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5529.306863] LDISKFS-fs (dm-2): recovery complete [ 5529.315446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5529.660353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.670579] Lustre: Skipped 4 previous similar messages [ 5532.408478] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5409 [ 5532.437913] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129 [ 5534.288143] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5569.873874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5572.385200] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5441 [ 5572.386858] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161 [ 5576.409478] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5587.820678] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5590.875042] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5632.415332] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 13:05:32 (1776186332) [ 5637.091606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5637.098531] Lustre: Skipped 3 previous similar messages [ 5657.656407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5657.767932] LustreError: 11-0: MGC192.168.204.134@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 5662.765564] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5663.310356] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5298 to 0x0:5313 [ 5663.312728] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5473 [ 5678.163746] Lustre: Failing over lustre-MDT0000 [ 5678.173368] Lustre: Skipped 5 previous similar messages [ 5678.510457] Lustre: server umount lustre-MDT0000 complete [ 5678.512262] Lustre: Skipped 5 previous similar messages [ 5678.565427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.578257] Lustre: Skipped 16 previous similar messages [ 5678.593539] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5678.629577] LustreError: Skipped 192 previous similar messages [ 5690.857287] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186384/real 1776186384] req@000000000f5c3703 x1862460514622016/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776186391 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 5690.876731] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5690.881966] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5690.888542] LustreError: Skipped 3 previous similar messages [ 5701.012665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.777855] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5708.785803] Lustre: Skipped 5 previous similar messages [ 5712.377544] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5712.897242] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5712.905728] Lustre: Skipped 5 previous similar messages [ 5712.931687] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5394 to 0x0:5505 [ 5712.932235] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5298 to 0x0:5345 [ 5722.385940] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5724.022434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5734.073589] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 13:07:14 (1776186434) [ 5767.599436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5769.785613] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5507 to 0x0:5537 [ 5769.801744] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:193 [ 5772.562862] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5782.551615] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5784.709783] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5797.660389] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 13:08:17 (1776186497) [ 5812.604436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.817709] Lustre: *** cfs_fail_loc=605, val=0*** [ 5812.827034] LustreError: 172432:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=000000007b95e8fa failed: rc = -95 [ 5812.861687] LustreError: 172432:0:(obd_config.c:774:class_setup()) setup MGS failed (-95) [ 5812.876414] LustreError: 172432:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95 [ 5812.886244] LustreError: 172432:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered [ 5812.898040] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5812.906706] LustreError: 172432:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 5812.948368] LustreError: 172432:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5823.825226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.295332] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5507 to 0x0:5569 [ 5833.299677] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5347 to 0x0:5377 [ 5833.624361] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5844.252438] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 13:09:04 (1776186544) [ 5852.819663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5882.103347] LDISKFS-fs (dm-0): recovery complete [ 5882.106341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.695926] Lustre: *** cfs_fail_loc=707, val=0*** [ 5888.147181] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5920.065778] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnected, waiting for 2 clients in recovery for 0:57 [ 5920.584768] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5391 to 0x0:5409 [ 5920.590662] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5582 to 0x0:5601 [ 5926.746498] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5928.220568] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5938.922102] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 13:10:39 (1776186639) [ 5971.226711] LustreError: 7728:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 5977.295155] LustreError: 7728:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5995.850944] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 13:11:36 (1776186696) [ 6027.082437] LustreError: 43043:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms [ 6033.120151] LustreError: 43043:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake [ 6042.181160] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 13:12:22 (1776186742) [ 6071.546455] LustreError: 9706:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 6076.551331] LustreError: 9706:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6078.993524] LustreError: 6262:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 6089.023749] LustreError: 6262:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6107.896754] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 13:13:28 (1776186808) [ 6172.272708] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 13:14:32 (1776186872) [ 6204.653545] LustreError: 7728:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6205.079350] LustreError: 7728:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6212.855274] LustreError: 44256:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6212.863687] LustreError: 44256:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 27 previous similar messages [ 6213.303136] LustreError: 44256:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6213.314710] LustreError: 44256:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 27 previous similar messages [ 6228.884790] LustreError: 20543:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6228.891929] LustreError: 20543:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 65 previous similar messages [ 6229.323148] LustreError: 20543:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6229.340068] LustreError: 20543:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 65 previous similar messages [ 6240.685214] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 13:15:40 (1776186940) [ 6276.725819] Lustre: DEBUG MARKER: phase 2 [ 6287.503252] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 13:16:27 (1776186987) [ 6371.682561] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 13:17:51 (1776187071) [ 6373.346444] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6375.179207] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 13:17:55 (1776187075) [ 6380.452893] Lustre: DEBUG MARKER: Started rundbench load pid=138680 ... [ 6391.797260] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6394.681155] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6396.855911] Lustre: Failing over lustre-MDT0000 [ 6396.861378] Lustre: Skipped 3 previous similar messages [ 6396.911543] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) [ 6397.140853] Lustre: server umount lustre-MDT0000 complete [ 6397.149484] Lustre: Skipped 4 previous similar messages [ 6399.458990] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6399.469327] LustreError: Skipped 3 previous similar messages [ 6399.475654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6399.491185] Lustre: Skipped 13 previous similar messages [ 6399.496447] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6399.516369] LustreError: Skipped 123 previous similar messages [ 6406.111374] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187100/real 1776187100] req@00000000a820994e x1862460514823168/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776187107 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 6406.147268] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6406.158060] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6406.165274] LustreError: Skipped 2 previous similar messages [ 6420.891688] LDISKFS-fs (dm-0): recovery complete [ 6420.897888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6423.523306] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459be038e to 0xca62691459be68b3 [ 6423.530940] Lustre: Skipped 4 previous similar messages [ 6423.540842] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 6423.552685] Lustre: Skipped 30 previous similar messages [ 6423.847605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6423.858143] Lustre: Skipped 6 previous similar messages [ 6423.886294] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6423.901506] Lustre: Skipped 7 previous similar messages [ 6424.353117] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6424.370039] Lustre: Skipped 3 previous similar messages [ 6427.965279] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6429.943531] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6429.948617] Lustre: Skipped 3 previous similar messages [ 6429.989318] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5703 to 0x0:5729 [ 6429.990553] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5492 to 0x0:5537 [ 6437.455327] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6439.087472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6451.092933] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6453.750023] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6479.813967] LDISKFS-fs (dm-1): recovery complete [ 6479.822053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.575771] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:197 to 0x280000400:225 [ 6485.577510] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:4 to 0x2c0000400:33 [ 6485.901108] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6497.567207] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6500.226120] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6534.519559] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 13:20:33 (1776187233) [ 6665.525148] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6678.621174] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 6680.578968] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.34@tcp (stopping) [ 6685.158325] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6714.070336] LDISKFS-fs (dm-1): recovery complete [ 6714.073574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6721.317674] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6723.152962] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:466 to 0x280000400:481 [ 6723.153631] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:274 to 0x2c0000400:289 [ 6732.064309] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6734.549810] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6818.471910] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 13:25:18 (1776187518) [ 6946.994163] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6959.339638] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 6961.294961] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) [ 6961.298117] Lustre: Skipped 3 previous similar messages [ 6986.315531] LDISKFS-fs (dm-0): recovery complete [ 6986.317734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6994.036687] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6996.635209] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6255 to 0x0:6273 [ 6996.638739] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6063 to 0x0:6081 [ 7003.268894] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7005.100414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7013.002405] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 13:28:33 (1776187713) [ 7143.382257] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7156.421467] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 7158.498572] Lustre: Failing over lustre-MDT0000 [ 7158.501124] Lustre: Skipped 3 previous similar messages [ 7158.627185] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) [ 7158.713839] Lustre: server umount lustre-MDT0000 complete [ 7158.722461] Lustre: Skipped 3 previous similar messages [ 7158.753205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7158.754944] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7158.774061] Lustre: Skipped 15 previous similar messages [ 7158.781890] LustreError: Skipped 111 previous similar messages [ 7171.039347] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187864/real 1776187864] req@000000005e73884a x1862460516727872/t0(0) o400->MGC192.168.204.134@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776187871 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 7171.080594] Lustre: 3351:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7171.088807] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7171.099230] LustreError: Skipped 1 previous similar message [ 7181.888977] LDISKFS-fs (dm-0): recovery complete [ 7181.892592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.460715] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459c7d647 to 0xca62691459cb45e6 [ 7188.477512] Lustre: Skipped 1 previous similar message [ 7188.488937] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 7188.496351] Lustre: Skipped 15 previous similar messages [ 7188.791626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7188.801042] Lustre: Skipped 3 previous similar messages [ 7188.859682] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7188.873307] Lustre: Skipped 3 previous similar messages [ 7189.805370] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7189.823845] Lustre: Skipped 3 previous similar messages [ 7194.356894] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7200.546977] Lustre: lustre-MDT0000: Recovery over after 0:11, of 2 clients 2 recovered and 0 were evicted. [ 7200.561572] Lustre: Skipped 3 previous similar messages [ 7200.613435] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6519 to 0x0:6561 [ 7200.617217] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6710 to 0x0:6753 [ 7206.986867] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7209.956968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7221.897900] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 13:32:02 (1776187922) [ 7235.836832] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7237.931446] Lustre: lustre-OST0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnecting [ 7237.944769] Lustre: Skipped 1 previous similar message [ 7238.891925] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7241.704311] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7241.717209] LustreError: Skipped 2 previous similar messages [ 7266.222388] LDISKFS-fs (dm-2): recovery complete [ 7266.229811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7268.596316] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:6817 [ 7268.602751] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:646 to 0x280000400:673 [ 7271.393656] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7282.183817] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7284.369892] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7295.745664] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 13:33:16 (1776187996) [ 7423.954258] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7431.857362] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7444.043653] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 7446.453450] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.34@tcp (stopping) [ 7451.457538] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188152 with bad export cookie 14583334079505843686 [ 7451.476404] LustreError: 6248:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7476.515688] LDISKFS-fs (dm-0): recovery complete [ 7476.518535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7488.705868] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7516.449683] LDISKFS-fs (dm-1): recovery complete [ 7516.452930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.907520] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7522.346034] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:455 to 0x2c0000400:481 [ 7522.349564] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:646 to 0x280000400:705 [ 7529.636576] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6603 to 0x0:6625 [ 7529.641372] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:6849 [ 7536.916054] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7538.965543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7541.015405] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7550.375620] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 13:37:30 (1776188250) [ 7558.498052] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7586.464355] LDISKFS-fs (dm-0): recovery complete [ 7586.467900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.443609] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7593.144064] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7596.862823] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 7596.919599] Lustre: 195659:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000001caca947 x1862460506974464/t335007449625(335007449625) o101->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:218/0 lens 648/3424 e 0 to 0 dl 1776188303 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7597.060336] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6627 to 0x0:6657 [ 7597.065176] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:6881 [ 7603.118432] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7604.876823] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7613.956723] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 13:38:34 (1776188314) [ 7621.401856] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7647.724593] LDISKFS-fs (dm-0): recovery complete [ 7647.731525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7654.721348] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7654.725526] LustreError: 195659:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000001ad25910 x1862460506974464/t335007449625(335007449625) o101->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:276/0 lens 648/600 e 0 to 0 dl 1776188361 ref 1 fl Interpret:/4/0 rc 301/0 job:'lfs.0' [ 7658.735097] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7661.886612] Lustre: lustre-MDT0000: Client e5918ffb-1455-4fdd-ac94-c09c5379639f (at 192.168.204.34@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 7661.910257] Lustre: 196434:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000fc7cbd31 x1862460506974464/t335007449625(335007449625) o101->e5918ffb-1455-4fdd-ac94-c09c5379639f@192.168.204.34@tcp:283/0 lens 648/3424 e 0 to 0 dl 1776188368 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7662.035266] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:6913 [ 7662.035468] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6659 to 0x0:6689 [ 7669.072426] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7671.210925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7681.851732] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 13:39:42 (1776188382) [ 7710.168473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7724.483576] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7725.597410] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6659 to 0x0:6721 [ 7732.236654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7733.677142] Lustre: lustre-OST0000: Denying connection for new client a22fada4-f15c-4972-9dcb-a08c6904c210 (at 192.168.204.34@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7733.703951] Lustre: Skipped 12 previous similar messages [ 7734.315722] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:646 to 0x280000400:737 [ 7734.323381] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:6945 [ 7736.877234] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7745.918319] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:40:46 (1776188446) [ 7746.926392] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7746.931892] LustreError: 8243:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000005ebc8957 x1862460517869952/t352187318279(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:368/0 lens 1056/4320 e 0 to 0 dl 1776188453 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7753.186803] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7753.310793] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7762.724426] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7762.741376] LustreError: Skipped 226 previous similar messages [ 7776.971511] LDISKFS-fs (dm-0): recovery complete [ 7776.978611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7787.680612] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7788.134817] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:6977 [ 7788.136614] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:6753 [ 7801.510632] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7803.768164] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7813.683661] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:41:53 (1776188513) [ 7814.703483] LustreError: 8242:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000003c77981b x1862460517893248/t356482285578(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:436/0 lens 2200/4320 e 0 to 0 dl 1776188521 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7821.472170] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7821.798891] Lustre: 196671:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776188515/real 1776188515] req@0000000016f41ad8 x1862460517893248/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2200/4320 e 0 to 1 dl 1776188522 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'osp_up0-1.0' [ 7821.833123] Lustre: 196671:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 26 previous similar messages [ 7821.840229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7821.854959] Lustre: Skipped 27 previous similar messages [ 7821.863656] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7821.876870] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 7821.882394] Lustre: Skipped 35 previous similar messages [ 7828.356356] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7830.162911] Lustre: Failing over lustre-MDT0001 [ 7830.167294] Lustre: Skipped 8 previous similar messages [ 7830.277260] Lustre: server umount lustre-MDT0001 complete [ 7830.289615] Lustre: Skipped 8 previous similar messages [ 7853.410823] LDISKFS-fs (dm-1): recovery complete [ 7853.414431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7853.649401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7853.658427] Lustre: Skipped 8 previous similar messages [ 7853.686346] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7853.693954] Lustre: Skipped 8 previous similar messages [ 7854.884902] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7854.888208] Lustre: Skipped 8 previous similar messages [ 7857.512842] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7858.679382] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7858.690664] Lustre: Skipped 8 previous similar messages [ 7858.731684] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:748 to 0x280000400:769 [ 7858.732819] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:492 to 0x2c0000400:513 [ 7866.411652] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7868.109604] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7878.844830] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:42:58 (1776188578) [ 7880.105177] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7880.111781] Lustre: Skipped 1 previous similar message [ 7880.115902] LustreError: 185270:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000005096f3d2 x1862460517914240/t356482285597(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:502/0 lens 2200/4320 e 0 to 0 dl 1776188587 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7887.334677] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7887.931662] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7894.800594] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7897.569470] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7897.585790] LustreError: Skipped 7 previous similar messages [ 7904.735221] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7904.752341] LustreError: Skipped 5 previous similar messages [ 7919.345452] LDISKFS-fs (dm-0): recovery complete [ 7919.349938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.121804] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459cf7d67 to 0xca62691459cf942e [ 7921.132456] Lustre: Skipped 5 previous similar messages [ 7924.851595] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7926.920514] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:6785 [ 7926.920694] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7009 [ 7934.081526] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7935.877102] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7963.052948] LDISKFS-fs (dm-1): recovery complete [ 7963.063079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7967.802614] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7968.813793] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:780 to 0x280000400:801 [ 7968.813896] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:524 to 0x2c0000400:545 [ 7977.669304] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7979.576766] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7993.053402] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:44:52 (1776188692) [ 8002.025945] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8006.789088] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8014.921502] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8021.098528] LustreError: 44521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188722 with bad export cookie 14583334079506125870 [ 8021.106494] LustreError: 44521:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8022.500623] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8022.507820] Lustre: Skipped 2 previous similar messages [ 8052.576563] LDISKFS-fs (dm-0): recovery complete [ 8052.590545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8062.935826] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8087.238965] LDISKFS-fs (dm-1): recovery complete [ 8087.246892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8092.128232] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8092.838692] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:812 to 0x280000400:833 [ 8092.839169] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:556 to 0x2c0000400:577 [ 8092.973633] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:6817 [ 8092.984772] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7041 [ 8102.201860] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8104.303489] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8105.685526] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8114.797419] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:46:54 (1776188814) [ 8116.219495] LustreError: 214172:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000765e15d4 x1862460507835072/t30064771142(0) o36->a22fada4-f15c-4972-9dcb-a08c6904c210@192.168.204.34@tcp:738/0 lens 560/448 e 0 to 0 dl 1776188823 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 8116.266825] LustreError: 214172:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 8123.676712] Lustre: lustre-MDT0001: Client a22fada4-f15c-4972-9dcb-a08c6904c210 (at 192.168.204.34@tcp) reconnecting [ 8123.689113] Lustre: 214173:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000f31241d5 x1862460507835072/t30064771142(0) o36->a22fada4-f15c-4972-9dcb-a08c6904c210@192.168.204.34@tcp:745/0 lens 560/448 e 0 to 0 dl 1776188830 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8127.175700] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8153.019705] LDISKFS-fs (dm-0): recovery complete [ 8153.023522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.476551] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8163.426036] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7073 [ 8163.426102] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:6849 [ 8172.258638] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8173.909718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8184.297851] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:48:04 (1776188884) [ 8185.497029] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8185.509481] Lustre: Skipped 2 previous similar messages [ 8192.816228] Lustre: lustre-MDT0001: Client a22fada4-f15c-4972-9dcb-a08c6904c210 (at 192.168.204.34@tcp) reconnecting [ 8192.847580] Lustre: 214174:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000023555f50 x1862460507852928/t30064771212(0) o36->a22fada4-f15c-4972-9dcb-a08c6904c210@192.168.204.34@tcp:59/0 lens 560/448 e 0 to 0 dl 1776188899 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8193.953402] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8220.877474] LDISKFS-fs (dm-1): recovery complete [ 8220.885454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.782601] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8226.322326] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:854 to 0x280000400:897 [ 8226.323291] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:598 to 0x2c0000400:641 [ 8236.796860] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8238.648799] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8247.832952] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:49:08 (1776188948) [ 8256.289987] Lustre: lustre-MDT0001: Client a22fada4-f15c-4972-9dcb-a08c6904c210 (at 192.168.204.34@tcp) reconnecting [ 8256.310583] Lustre: 214173:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000001a9193fd x1862460507869504/t34359738438(0) o36->a22fada4-f15c-4972-9dcb-a08c6904c210@192.168.204.34@tcp:123/0 lens 560/448 e 0 to 0 dl 1776188963 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8259.997760] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8267.311328] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8292.104800] LDISKFS-fs (dm-0): recovery complete [ 8292.107669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8302.386799] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8302.763126] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7105 [ 8302.770687] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:6881 [ 8312.466609] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8314.404939] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8345.526992] LDISKFS-fs (dm-1): recovery complete [ 8345.534886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8351.307667] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:908 to 0x280000400:929 [ 8351.319781] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:652 to 0x2c0000400:673 [ 8351.325173] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8362.429513] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8364.382566] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8374.818602] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:51:15 (1776189075) [ 8375.941846] LustreError: 214174:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000006b96236f x1862460507891904/t38654705734(0) o36->a22fada4-f15c-4972-9dcb-a08c6904c210@192.168.204.34@tcp:242/0 lens 560/448 e 0 to 0 dl 1776189082 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 8375.963758] LustreError: 214174:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 2 previous similar messages [ 8382.244682] Lustre: lustre-MDT0001: Client a22fada4-f15c-4972-9dcb-a08c6904c210 (at 192.168.204.34@tcp) reconnecting [ 8382.263620] Lustre: 217308:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000876f914b x1862460507891904/t38654705734(0) o36->a22fada4-f15c-4972-9dcb-a08c6904c210@192.168.204.34@tcp:249/0 lens 560/448 e 0 to 0 dl 1776189089 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8386.537516] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8393.345894] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8397.285517] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8397.298723] LustreError: Skipped 263 previous similar messages [ 8398.901842] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189099 with bad export cookie 14583334079506139562 [ 8398.914036] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8424.662761] LDISKFS-fs (dm-0): recovery complete [ 8424.665756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8431.974289] Lustre: MGC192.168.204.134@tcp: Connection restored to (at 0@lo) [ 8431.987408] Lustre: Skipped 36 previous similar messages [ 8437.080325] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8461.806267] LDISKFS-fs (dm-1): recovery complete [ 8461.817709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8462.166928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8462.170756] Lustre: Skipped 9 previous similar messages [ 8462.196777] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8462.213046] Lustre: Skipped 9 previous similar messages [ 8463.961743] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8463.975590] Lustre: Skipped 9 previous similar messages [ 8467.554747] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8468.416227] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8468.427352] Lustre: Skipped 8 previous similar messages [ 8468.489057] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:684 to 0x2c0000400:705 [ 8468.492150] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:940 to 0x280000400:961 [ 8468.579790] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:6913 [ 8468.580945] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7137 [ 8478.540769] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8480.573065] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8482.190412] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8490.729516] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:53:11 (1776189191) [ 8498.437199] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8499.167181] Lustre: 227001:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776189193/real 1776189193] req@00000000f0008227 x1862460518092800/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 640/4320 e 0 to 1 dl 1776189200 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'osp_up0-1.0' [ 8499.208230] Lustre: 227001:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 8499.220322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.234508] Lustre: Skipped 38 previous similar messages [ 8499.243263] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8500.173324] Lustre: Failing over lustre-MDT0001 [ 8500.177108] Lustre: Skipped 10 previous similar messages [ 8500.310797] Lustre: server umount lustre-MDT0001 complete [ 8500.313613] Lustre: Skipped 10 previous similar messages [ 8523.549815] LDISKFS-fs (dm-1): recovery complete [ 8523.564427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8528.558837] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8529.440641] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:737 [ 8529.441587] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:993 [ 8537.950472] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8539.311592] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8547.294168] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:54:07 (1776189247) [ 8555.301819] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8555.493645] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8567.263193] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8567.273253] LustreError: Skipped 4 previous similar messages [ 8580.369574] LDISKFS-fs (dm-0): recovery complete [ 8580.372685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8584.677237] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459cfd883 to 0xca62691459cfeb8b [ 8584.693599] Lustre: Skipped 4 previous similar messages [ 8590.438714] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:6945 [ 8590.443876] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7169 [ 8590.878816] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8600.471897] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8601.845877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8611.092399] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:55:11 (1776189311) [ 8619.493890] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8620.699714] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8628.382245] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8634.848483] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8634.855312] LustreError: Skipped 8 previous similar messages [ 8652.577427] LDISKFS-fs (dm-0): recovery complete [ 8652.581096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8664.163848] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8664.229573] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:6977 [ 8664.233806] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7201 [ 8676.226704] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8678.448542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8706.740759] LDISKFS-fs (dm-1): recovery complete [ 8706.749615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8712.044914] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8712.263424] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:769 [ 8723.881909] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8725.558122] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8734.553075] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:57:15 (1776189435) [ 8735.799568] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8735.805848] Lustre: Skipped 5 previous similar messages [ 8742.895623] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8743.549276] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8751.361135] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8757.455245] LustreError: 9227:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189458 with bad export cookie 14583334079506149887 [ 8757.474716] LustreError: 9227:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8758.245735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8758.255429] Lustre: Skipped 4 previous similar messages [ 8762.273053] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8762.280479] Lustre: Skipped 3 previous similar messages [ 8788.963311] LDISKFS-fs (dm-0): recovery complete [ 8788.967219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8797.228715] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8820.016156] LDISKFS-fs (dm-1): recovery complete [ 8820.023751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8825.309750] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8826.480860] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7233 [ 8826.485276] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:7009 [ 8826.588947] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:801 [ 8826.592156] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1025 [ 8836.999399] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8839.107549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8841.027927] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8850.326208] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:59:10 (1776189550) [ 8858.915369] Lustre: lustre-MDT0001: Client a22fada4-f15c-4972-9dcb-a08c6904c210 (at 192.168.204.34@tcp) reconnecting [ 8858.935350] Lustre: 237802:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000004449cefc x1862460507949696/t55834574850(0) o36->a22fada4-f15c-4972-9dcb-a08c6904c210@192.168.204.34@tcp:725/0 lens 496/2888 e 0 to 0 dl 1776189565 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0' [ 8860.242385] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8887.108580] LDISKFS-fs (dm-0): recovery complete [ 8887.111732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8895.774423] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8896.111045] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:7041 [ 8896.111394] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7265 [ 8906.331768] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8908.083342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8917.699167] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 14:00:17 (1776189617) [ 8919.115074] LustreError: 237802:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000006c1c117f x1862460507958464/t55834574852(0) o36->a22fada4-f15c-4972-9dcb-a08c6904c210@192.168.204.34@tcp:31/0 lens 496/456 e 0 to 0 dl 1776189626 ref 1 fl Interpret:/0/0 rc 0/0 job:'rmdir.0' [ 8919.151643] LustreError: 237802:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 5 previous similar messages [ 8926.265493] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8951.575912] LDISKFS-fs (dm-1): recovery complete [ 8951.580754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8956.605566] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8957.478911] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1057 [ 8957.479079] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:833 [ 8966.553964] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8968.465619] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8976.505415] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 14:01:16 (1776189676) [ 8985.877506] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8992.856181] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8998.372546] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8998.387868] LustreError: Skipped 219 previous similar messages [ 9018.358029] LDISKFS-fs (dm-0): recovery complete [ 9018.361590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.822980] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7297 [ 9028.824487] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:7073 [ 9029.054404] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9040.082231] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9041.918477] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9070.004658] LDISKFS-fs (dm-1): recovery complete [ 9070.012570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9070.396090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9070.414472] Lustre: Skipped 9 previous similar messages [ 9070.475459] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 9070.478034] Lustre: Skipped 9 previous similar messages [ 9070.882727] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9070.885310] Lustre: Skipped 9 previous similar messages [ 9075.427790] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9075.694275] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 9075.709431] Lustre: Skipped 46 previous similar messages [ 9075.778557] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9075.803505] Lustre: Skipped 10 previous similar messages [ 9075.861403] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1089 [ 9075.864676] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:865 [ 9086.063783] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9088.256674] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9099.137085] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 14:03:18 (1776189798) [ 9107.777432] Lustre: lustre-MDT0001: Client a22fada4-f15c-4972-9dcb-a08c6904c210 (at 192.168.204.34@tcp) reconnecting [ 9107.789870] Lustre: Skipped 2 previous similar messages [ 9110.152902] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9118.877223] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9121.091077] Lustre: Failing over lustre-MDT0000 [ 9121.097327] Lustre: Skipped 9 previous similar messages [ 9121.382552] Lustre: server umount lustre-MDT0000 complete [ 9121.386720] Lustre: Skipped 9 previous similar messages [ 9121.762761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9121.790749] Lustre: Skipped 37 previous similar messages [ 9125.749945] LustreError: 106211:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776189826 with bad export cookie 14583334079506157153 [ 9125.764347] LustreError: 106211:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9133.919335] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776189827/real 1776189827] req@00000000224cfb7e x1862460518275712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1776189834 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 9133.974853] Lustre: 3350:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 9149.541788] LDISKFS-fs (dm-0): recovery complete [ 9149.550768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9163.648709] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9187.765981] LDISKFS-fs (dm-1): recovery complete [ 9187.770183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.875703] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9193.979236] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6723 to 0x0:7105 [ 9193.985252] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6796 to 0x0:7329 [ 9194.063730] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:897 [ 9194.063963] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1121 [ 9203.591989] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9205.303125] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9207.166427] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9216.799413] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 14:05:17 (1776189917) [ 9218.881488] Lustre: 251688:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a22fada4-f15c-4972-9dcb-a08c6904c210 at adminstrative request [ 9228.243870] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 14:05:28 (1776189928) [ 9246.623278] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9246.630640] LustreError: Skipped 5 previous similar messages [ 9257.365307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9263.077371] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459d017ad to 0xca62691459d03d62 [ 9263.088877] Lustre: Skipped 5 previous similar messages [ 9268.485827] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9268.998890] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7381 to 0x0:7425 [ 9268.999318] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7156 to 0x0:7201 [ 9279.038438] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9281.286830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9290.386200] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 14:06:30 (1776189990) [ 9309.676272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9309.687660] Lustre: Skipped 1 previous similar message [ 9329.456568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9331.512333] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1153 [ 9331.516987] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7526 to 0x0:7553 [ 9334.536970] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9344.343935] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9346.213503] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9356.929665] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 14:07:37 (1776190057) [ 9371.015885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9371.184734] LustreError: 11-0: MGC192.168.204.134@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 9371.201317] LustreError: Skipped 8 previous similar messages [ 9376.325166] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9376.788778] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7156 to 0x0:7233 [ 9376.791246] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7526 to 0x0:7585 [ 9386.582369] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 14:08:06 (1776190086) [ 9395.605734] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9423.611718] LDISKFS-fs (dm-2): recovery complete [ 9423.618090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9425.477940] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7587 to 0x0:7617 [ 9425.486102] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1185 [ 9429.490684] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9440.929251] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9442.844404] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9452.729790] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 14:09:12 (1776190152) [ 9461.932293] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9489.982583] LDISKFS-fs (dm-2): recovery complete [ 9489.989654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9491.917222] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.34@tcp inode [0x20002f201:0x5:0x0] object 0x0:7618 extent [0-4194303]: client csum 8f2034ce, server csum abbe1bf5 [ 9492.139533] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7619 to 0x0:7649 [ 9492.183466] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1217 [ 9495.940039] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9508.244995] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9510.757472] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9519.728311] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 14:10:20 (1776190220) [ 9528.752195] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9535.349448] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9547.178962] LustreError: 106211:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190248 with bad export cookie 14583334079506183837 [ 9547.200846] LustreError: 106211:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9569.941861] LDISKFS-fs (dm-0): recovery complete [ 9569.946260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9585.790458] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9586.871674] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7156 to 0x0:7265 [ 9601.510780] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9601.531194] LustreError: Skipped 245 previous similar messages [ 9609.206122] LDISKFS-fs (dm-2): recovery complete [ 9609.221078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9610.996795] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1249 [ 9614.010821] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9631.993809] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 14:12:12 (1776190332) [ 9665.746465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9674.432098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9674.441725] Lustre: Skipped 9 previous similar messages [ 9674.492762] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9674.505819] Lustre: Skipped 9 previous similar messages [ 9674.539942] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9674.551552] Lustre: Skipped 9 previous similar messages [ 9678.899521] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9679.849771] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to (at 0@lo) [ 9679.859726] Lustre: Skipped 32 previous similar messages [ 9679.913830] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9679.922792] Lustre: Skipped 9 previous similar messages [ 9680.018362] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7156 to 0x0:7297 [ 9687.153331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9691.944743] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9694.507884] Lustre: lustre-OST0000: Denying connection for new client 266105b3-eb5e-4852-8db3-8cceab49e0c3 (at 192.168.204.34@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 9699.615503] Lustre: lustre-OST0000: Denying connection for new client 266105b3-eb5e-4852-8db3-8cceab49e0c3 (at 192.168.204.34@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 9704.753433] Lustre: lustre-OST0000: Denying connection for new client 266105b3-eb5e-4852-8db3-8cceab49e0c3 (at 192.168.204.34@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 9709.853512] Lustre: lustre-OST0000: Denying connection for new client 266105b3-eb5e-4852-8db3-8cceab49e0c3 (at 192.168.204.34@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 9720.107154] Lustre: lustre-OST0000: Denying connection for new client 266105b3-eb5e-4852-8db3-8cceab49e0c3 (at 192.168.204.34@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 9720.137196] Lustre: Skipped 1 previous similar message [ 9740.578055] Lustre: lustre-OST0000: Denying connection for new client 266105b3-eb5e-4852-8db3-8cceab49e0c3 (at 192.168.204.34@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 9740.595717] Lustre: Skipped 3 previous similar messages [ 9759.004603] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9759.025134] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9759.080540] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7700 to 0x0:7721 [ 9759.086315] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1281 [ 9763.699941] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 58 sec [ 9772.587902] Lustre: DEBUG MARKER: free_before: 7646580 free_after: 7646580 [ 9779.125757] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 14:14:39 (1776190479) [ 9783.623948] Lustre: Failing over lustre-OST0001 [ 9783.626304] Lustre: Skipped 10 previous similar messages [ 9783.654995] Lustre: server umount lustre-OST0001 complete [ 9783.656759] Lustre: Skipped 10 previous similar messages [ 9786.849883] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9786.871179] Lustre: Skipped 27 previous similar messages [ 9803.883401] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9806.241415] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:929 [ 9806.250722] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7300 to 0x0:7329 [ 9809.705769] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9821.845627] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 14:15:22 (1776190522) [ 9846.928358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9848.319491] LustreError: 268367:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9848.329683] LustreError: 268367:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 9849.121161] Lustre: *** cfs_fail_loc=715, val=0*** [ 9852.048658] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9855.455134] Lustre: 3349:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776190549/real 1776190549] req@000000005f656286 x1862460518471104/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1776190556 ref 1 fl Rpc:XQr/c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' [ 9855.492461] Lustre: 3349:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 9855.497795] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:58 [ 9855.532839] Lustre: Skipped 1 previous similar message [ 9856.543203] Lustre: *** cfs_fail_loc=715, val=0*** [ 9856.546453] Lustre: Skipped 2 previous similar messages [ 9862.628280] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:51 [ 9862.642513] Lustre: Skipped 2 previous similar messages [ 9863.647451] Lustre: *** cfs_fail_loc=715, val=0*** [ 9863.653702] Lustre: Skipped 2 previous similar messages [ 9869.792437] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:44 [ 9869.802850] Lustre: Skipped 2 previous similar messages [ 9870.816292] Lustre: *** cfs_fail_loc=715, val=0*** [ 9870.821196] Lustre: Skipped 2 previous similar messages [ 9876.960207] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:37 [ 9876.967156] Lustre: Skipped 2 previous similar messages [ 9877.988752] Lustre: *** cfs_fail_loc=715, val=0*** [ 9877.996537] Lustre: Skipped 2 previous similar messages [ 9883.111282] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:30 [ 9883.125735] Lustre: Skipped 2 previous similar messages [ 9888.416377] LustreError: 268367:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [ 9888.440728] LustreError: 268367:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 9888.494694] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7725 to 0x0:7753 [ 9888.497358] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1313 [ 9897.396574] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9899.218735] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9909.719431] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 14:16:49 (1776190609) [ 9925.071927] LustreError: 166-1: MGC192.168.204.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9925.092145] LustreError: Skipped 3 previous similar messages [ 9935.216622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9942.499150] Lustre: Evicted from MGS (at 192.168.204.134@tcp) after server handle changed from 0xca62691459d09d6a to 0xca62691459d0aedc [ 9942.505140] Lustre: Skipped 3 previous similar messages [ 9945.375325] Lustre: *** cfs_fail_loc=715, val=0*** [ 9945.377630] Lustre: Skipped 5 previous similar messages [ 9947.403223] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9948.162543] LustreError: 269933:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [ 9951.524254] Lustre: lustre-MDT0000: Client 266105b3-eb5e-4852-8db3-8cceab49e0c3 (at 192.168.204.34@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 9951.537332] Lustre: Skipped 1 previous similar message [ 9955.297142] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9963.487530] Lustre: *** cfs_fail_loc=715, val=0*** [ 9963.493929] Lustre: Skipped 4 previous similar messages [ 9969.631645] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9969.639605] Lustre: Skipped 1 previous similar message [ 9973.020443] Lustre: lustre-MDT0000: Client 266105b3-eb5e-4852-8db3-8cceab49e0c3 (at 192.168.204.34@tcp) reconnected, waiting for 2 clients in recovery for 0:36 [ 9973.036860] Lustre: Skipped 2 previous similar messages [ 9990.111748] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9990.126152] Lustre: Skipped 2 previous similar messages [ 9998.304070] Lustre: *** cfs_fail_loc=715, val=0*** [ 9998.308172] Lustre: Skipped 9 previous similar messages [10007.838827] Lustre: lustre-MDT0000: Client 266105b3-eb5e-4852-8db3-8cceab49e0c3 (at 192.168.204.34@tcp) reconnected, waiting for 2 clients in recovery for 0:02 [10007.850111] Lustre: Skipped 4 previous similar messages [10015.010137] Lustre: lustre-MDT0000: Recovery already passed deadline 0:05. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [10021.151426] Lustre: lustre-MDT0000: Recovery already passed deadline 0:11. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [10025.951907] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10025.972502] Lustre: Skipped 4 previous similar messages [10028.199128] LustreError: 269933:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [10028.228226] Lustre: 269933:0:(ldlm_lib.c:2849:target_recovery_thread()) too long recovery - read logs [10028.243352] LustreError: dumping log to /tmp/lustre-log.1776190729.269933 [10028.384113] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7764 to 0x0:7785 [10028.386983] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7340 to 0x0:7361 [10036.003826] Lustre: DEBUG MARKER: oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10037.794107] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10049.721843] Lustre: DEBUG MARKER: == replay-single test complete, duration 9848 sec ======== 14:19:09 (1776190749) [10064.355670] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10064.374068] LustreError: Skipped 5 previous similar messages [10064.387647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10069.987845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10069.995850] Lustre: Skipped 6 previous similar messages [10079.188788] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776190780 with bad export cookie 14583334079506198236 [10079.208958] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [10114.329501] Lustre: DEBUG MARKER: oleg434-server.virtnet: executing unload_modules_local [10116.664969] Key type lgssc unregistered [10116.946416] LNet: 273138:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10116.955039] LNet: Removed LNI 192.168.204.134@tcp [10117.802521] Key type .llcrypt unregistered [10117.805716] Key type ._llcrypt unregistered