[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 469820457 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002393] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004022] kvm-guest: setup PV IPIs [ 0.007477] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009020] pid_max: default: 32768 minimum: 301 [ 0.010179] LSM: Security Framework initializing [ 0.011087] Yama: becoming mindful. [ 0.012065] SELinux: Initializing. [ 0.013109] *** VALIDATE selinux *** [ 0.021425] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026506] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028036] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029141] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030167] *** VALIDATE tmpfs *** [ 0.032544] *** VALIDATE proc *** [ 0.034128] *** VALIDATE cgroup *** [ 0.035025] *** VALIDATE cgroup2 *** [ 0.037383] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038248] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039020] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040058] Spectre V2 : User space: Vulnerable [ 0.041025] Speculative Store Bypass: Vulnerable [ 0.045490] debug: unmapping init [mem 0xffffffff84659000-0xffffffff84660fff] [ 0.048419] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050560] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051043] ... version: 2 [ 0.052026] ... bit width: 48 [ 0.053032] ... generic registers: 4 [ 0.054033] ... value mask: 0000ffffffffffff [ 0.055033] ... max period: 00007fffffffffff [ 0.056031] ... fixed-purpose events: 3 [ 0.057042] ... event mask: 000000070000000f [ 0.058509] rcu: Hierarchical SRCU implementation. [ 0.060624] smp: Bringing up secondary CPUs ... [ 0.061685] x86: Booting SMP configuration: [ 0.062053] .... node #0, CPUs: #1 #2 #3 [ 0.067626] smp: Brought up 1 node, 4 CPUs [ 0.069034] smpboot: Max logical packages: 1 [ 0.070029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.252195] node 0 deferred pages initialised in 178ms [ 0.256556] devtmpfs: initialized [ 0.257615] x86/mm: Memory block size: 128MB [ 0.261717] gcov: version magic: 0x41383552 [ 0.264042] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.265153] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.266565] pinctrl core: initialized pinctrl subsystem [ 0.267409] [ 0.268025] ************************************************************* [ 0.269023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.270033] ** ** [ 0.271036] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.272030] ** ** [ 0.273030] ** This means that this kernel is built to expose internal ** [ 0.274029] ** IOMMU data structures, which may compromise security on ** [ 0.275030] ** your system. ** [ 0.276030] ** ** [ 0.277029] ** If you see this message and you are not debugging the ** [ 0.278031] ** kernel, report this immediately to your vendor! ** [ 0.279029] ** ** [ 0.280029] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.281034] ************************************************************* [ 0.283101] NET: Registered protocol family 16 [ 0.284927] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.285108] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.286114] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.288069] cpuidle: using governor menu [ 0.289000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.294568] PCI: Using configuration type 1 for base access [ 0.297129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.308513] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.311029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.315112] cryptd: max_cpu_qlen set to 1000 [ 0.317290] ACPI: Added _OSI(Module Device) [ 0.318020] ACPI: Added _OSI(Processor Device) [ 0.319018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.320016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.324564] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.327594] ACPI: Interpreter enabled [ 0.328111] ACPI: PM: (supports S0 S3 S4 S5) [ 0.329019] ACPI: Using IOAPIC for interrupt routing [ 0.332099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.335550] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.348180] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.350086] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.353019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.356163] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.363619] acpiphp: Slot [2] registered [ 0.365214] acpiphp: Slot [5] registered [ 0.367197] acpiphp: Slot [6] registered [ 0.368110] acpiphp: Slot [7] registered [ 0.370162] acpiphp: Slot [8] registered [ 0.371104] acpiphp: Slot [9] registered [ 0.373109] acpiphp: Slot [10] registered [ 0.375146] acpiphp: Slot [3] registered [ 0.377129] acpiphp: Slot [4] registered [ 0.379314] acpiphp: Slot [11] registered [ 0.381134] acpiphp: Slot [12] registered [ 0.383159] acpiphp: Slot [13] registered [ 0.385112] acpiphp: Slot [14] registered [ 0.386092] acpiphp: Slot [15] registered [ 0.388121] acpiphp: Slot [16] registered [ 0.390119] acpiphp: Slot [17] registered [ 0.392150] acpiphp: Slot [18] registered [ 0.394136] acpiphp: Slot [19] registered [ 0.396278] acpiphp: Slot [20] registered [ 0.398266] acpiphp: Slot [21] registered [ 0.400162] acpiphp: Slot [22] registered [ 0.402169] acpiphp: Slot [23] registered [ 0.404285] acpiphp: Slot [24] registered [ 0.405319] acpiphp: Slot [25] registered [ 0.408170] acpiphp: Slot [26] registered [ 0.409270] acpiphp: Slot [27] registered [ 0.411173] acpiphp: Slot [28] registered [ 0.413155] acpiphp: Slot [29] registered [ 0.415178] acpiphp: Slot [30] registered [ 0.417207] acpiphp: Slot [31] registered [ 0.419099] PCI host bridge to bus 0000:00 [ 0.420033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.423042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.426053] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.429050] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.433044] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.437030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.439289] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.443250] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.446407] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.457978] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.463119] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.466023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.469020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.473023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.477209] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.479881] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.483064] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.486898] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.493025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.508032] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.515024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.522908] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.529016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.536024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.555017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.568135] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.576024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.583028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.600020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.610326] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.621027] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.629018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.649016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.660029] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.668015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.676018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.691022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.702643] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.714019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.720018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.739173] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.749951] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.761018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.767018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.787017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.796481] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.798345] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.801346] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.803377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.806193] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.810008] iommu: Default domain type: Passthrough [ 0.812797] SCSI subsystem initialized [ 0.814173] ACPI: bus type USB registered [ 0.816124] usbcore: registered new interface driver usbfs [ 0.818133] usbcore: registered new interface driver hub [ 0.821111] usbcore: registered new device driver usb [ 0.823198] pps_core: LinuxPPS API ver. 1 registered [ 0.824009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.828066] PTP clock support registered [ 0.830062] EDAC MC: Ver: 3.0.0 [ 0.832134] PCI: Using ACPI for IRQ routing [ 0.834923] NetLabel: Initializing [ 0.836012] NetLabel: domain hash size = 128 [ 0.837010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.840100] NetLabel: unlabeled traffic allowed by default [ 0.842124] vgaarb: loaded [ 0.844229] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.845014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.852829] clocksource: Switched to clocksource kvm-clock [ 0.966512] VFS: Disk quotas dquot_6.6.0 [ 0.967850] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.970488] *** VALIDATE ramfs *** [ 0.971881] *** VALIDATE hugetlbfs *** [ 0.973465] pnp: PnP ACPI init [ 0.976033] pnp: PnP ACPI: found 6 devices [ 0.991935] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.995611] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.997742] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.999498] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.001514] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.003340] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.005364] NET: Registered protocol family 2 [ 1.007692] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.011620] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.015880] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.022564] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.026637] TCP: Hash tables configured (established 65536 bind 65536) [ 1.030267] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.034504] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.037417] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.040923] NET: Registered protocol family 1 [ 1.044745] RPC: Registered named UNIX socket transport module. [ 1.048632] RPC: Registered udp transport module. [ 1.051562] RPC: Registered tcp transport module. [ 1.054270] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.057849] NET: Registered protocol family 44 [ 1.060246] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.063675] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.066068] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.069152] PCI: CLS 0 bytes, default 64 [ 1.071957] Unpacking initramfs... [ 2.530739] debug: unmapping init [mem 0xffff8b6c3cc54000-0xffff8b6c3ffbffff] [ 2.534692] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.537453] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.540461] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.024442] Initialise system trusted keyrings [ 3.026773] Key type blacklist registered [ 3.039552] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.052318] zbud: loaded [ 3.055357] *** VALIDATE nfs *** [ 3.056486] *** VALIDATE nfs4 *** [ 3.057793] pstore: using deflate compression [ 3.060480] Platform Keyring initialized [ 3.172521] NET: Registered protocol family 38 [ 3.174361] Key type asymmetric registered [ 3.175948] Asymmetric key parser 'x509' registered [ 3.177478] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.180415] io scheduler mq-deadline registered [ 3.182115] io scheduler kyber registered [ 3.183892] io scheduler bfq registered [ 3.185678] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.188810] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.191752] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.194555] ACPI: Power Button [PWRF] [ 3.200518] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.206746] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.219806] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.227745] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.249194] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.278634] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.307972] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.314450] Non-volatile memory driver v1.3 [ 3.316504] Linux agpgart interface v0.103 [ 3.354521] virtio_blk virtio1: [vda] 68040 512-byte logical blocks (34.8 MB/33.2 MiB) [ 3.359673] vda: detected capacity change from 0 to 34836480 [ 3.377717] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.380756] vdb: detected capacity change from 0 to 1073741824 [ 3.399434] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.403668] vdc: detected capacity change from 0 to 2621440000 [ 3.424415] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.427875] vdd: detected capacity change from 0 to 2621440000 [ 3.442932] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.446344] vde: detected capacity change from 0 to 4294967296 [ 3.462720] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.466026] vdf: detected capacity change from 0 to 4294967296 [ 3.474946] libphy: Fixed MDIO Bus: probed [ 3.484173] usbcore: registered new interface driver usbserial_generic [ 3.487099] usbserial: USB Serial support registered for generic [ 3.489367] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.494273] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.496109] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.498882] mousedev: PS/2 mouse device common for all mice [ 3.501153] rtc_cmos 00:05: RTC can wake from S4 [ 3.505840] rtc_cmos 00:05: registered as rtc0 [ 3.510445] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.511263] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.513593] intel_pstate: CPU model not supported [ 3.520809] hid: raw HID events driver (C) Jiri Kosina [ 3.522696] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.524194] usbcore: registered new interface driver usbhid [ 3.531382] usbhid: USB HID core driver [ 3.533220] drop_monitor: Initializing network drop monitor service [ 3.536042] Initializing XFRM netlink socket [ 3.538143] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.538229] NET: Registered protocol family 10 [ 3.547176] Segment Routing with IPv6 [ 3.549208] NET: Registered protocol family 17 [ 3.551575] mpls_gso: MPLS GSO support [ 3.558489] RAS: Correctable Errors collector initialized. [ 3.560846] AVX version of gcm_enc/dec engaged. [ 3.563223] AES CTR mode by8 optimization enabled [ 3.639883] sched_clock: Marking stable (3639858614, 0)->(4796906807, -1157048193) [ 3.643831] registered taskstats version 1 [ 3.646419] Loading compiled-in X.509 certificates [ 3.648906] zswap: loaded using pool lzo/zbud [ 3.676342] Key type big_key registered [ 3.693659] Key type encrypted registered [ 3.695718] ima: No TPM chip found, activating TPM-bypass! [ 3.698359] ima: Allocated hash algorithm: sha1 [ 3.700629] ima: No architecture policies found [ 3.702848] evm: Initialising EVM extended attributes: [ 3.705205] evm: security.selinux [ 3.706808] evm: security.ima [ 3.708274] evm: security.capability [ 3.709819] evm: HMAC attrs: 0x1 [ 3.712927] rtc_cmos 00:05: setting system clock to 2026-04-14 14:54:49 UTC (1776178489) [ 3.721357] debug: unmapping init [mem 0xffffffff85603000-0xffffffff857fffff] [ 3.725514] debug: unmapping init [mem 0xffffffff84382000-0xffffffff84658fff] [ 3.738111] Write protecting the kernel read-only data: 28672k [ 3.742816] debug: unmapping init [mem 0xffffffff82a03000-0xffffffff82bfffff] [ 3.746043] debug: unmapping init [mem 0xffffffff83314000-0xffffffff833fffff] [ 3.780779] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.786816] systemd[1]: Detected virtualization kvm. [ 3.788034] systemd[1]: Detected architecture x86-64. [ 3.789202] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.818896] systemd[1]: No hostname configured. [ 3.820278] systemd[1]: Set hostname to . [ 3.822345] random: systemd: uninitialized urandom read (16 bytes read) [ 3.824549] systemd[1]: Initializing machine ID from random generator. [ 3.873582] random: ln: uninitialized urandom read (6 bytes read) [ 3.972170] random: systemd: uninitialized urandom read (16 bytes read) [ 3.975403] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.980128] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.987973] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.629467] device-mapper: uevent: version 1.0.3 [ 4.631658] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ 4.984159] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.418346] virtio_net virtio0 ens2: renamed from eth0 [ 5.493190] scsi host0: ata_piix [ 5.498085] scsi host1: ata_piix [ 5.504342] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.508107] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.093302] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.215705] random: crng init done [ 10.217472] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.670695] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.816194] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.081499] SELinux: Disabled at runtime. [ 12.146992] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.158396] systemd[1]: Detected virtualization kvm. [ 12.160623] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.638235] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.641252] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.646688] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.650148] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.653825] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.664865] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.674659] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ 12.799250] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.108981] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.394886] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.429638] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.578358] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.595296] EDAC sbridge: Ver: 1.1.2 [ 15.084644] Key type dns_resolver registered [ 15.389354] NFS: Registering the id_resolver key type [ 15.391060] Key type id_resolver registered [ 15.392700] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg430-server login: [ 67.913895] libcfs: loading out-of-tree module taints kernel. [ 68.015135] alg: No test for adler32 (adler32-zlib) [ 68.774867] Key type ._llcrypt registered [ 68.782227] Key type .llcrypt registered [ 68.970956] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 89.583534] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 92.099605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 93.137702] Lustre: Lustre: Build Version: 2.15.8_13_g05ba8a9 [ 94.184614] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 94.189748] LNet: Accept secure, port 988 [ 96.088835] Key type lgssc registered [ 99.058109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.111115] hrtimer: interrupt took 2399910 ns [ 130.601255] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 135.880410] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 147.178903] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 153.850205] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.870540] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.775261] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 189.758652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 189.881805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.216414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 191.246992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 191.352850] Lustre: lustre-MDT0000: new disk, initializing [ 191.452324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.485880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.037692] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 209.387210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.523614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.636585] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 209.671267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 209.685046] Lustre: Skipped 1 previous similar message [ 209.907314] Lustre: lustre-MDT0001: new disk, initializing [ 209.998958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.070027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 210.106464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.743220] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 233.567661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 233.649920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 233.656564] Lustre: lustre-OST0000-osd: enabled 'large_dir' feature on device /dev/mapper/ost1_flakey [ 233.872520] Lustre: lustre-OST0000: new disk, initializing [ 233.876971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 233.960091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.324837] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 243.209453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 243.216324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 248.936326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 249.002876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.010734] Lustre: lustre-OST0001-osd: enabled 'large_dir' feature on device /dev/mapper/ost2_flakey [ 249.089380] Lustre: lustre-OST0001: new disk, initializing [ 249.094276] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 249.150633] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.114756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.125352] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 253.000075] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 262.149903] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 267.521144] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 276.097277] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing check_logdir /tmp/testlogs/ [ 280.338682] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing yml_node [ 286.029586] Lustre: DEBUG MARKER: Client: 2.15.8.13 [ 289.698802] Lustre: DEBUG MARKER: MDS: 2.15.8.13 [ 293.023653] Lustre: DEBUG MARKER: OSS: 2.15.8.13 [ 295.440275] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Apr 14 10:59:37 EDT 2026 [ 307.606347] Lustre: DEBUG MARKER: excepting tests: 59 [ 313.790841] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing check_config_client /mnt/lustre [ 332.547760] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 335.710774] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 338.699067] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 341.792483] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 11:00:25 (1776178825) [ 349.373573] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 351.503667] Lustre: Failing over lustre-MDT0000 [ 351.744033] Lustre: server umount lustre-MDT0000 complete [ 352.743370] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.752270] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.766953] Lustre: Skipped 1 previous similar message [ 352.783547] LustreError: Skipped 3 previous similar messages [ 355.460965] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.856609] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.879380] LustreError: Skipped 3 previous similar messages [ 359.839111] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776178838/real 1776178838] req@00000000b7da9226 x1862458229704448/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776178845 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 359.857901] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 359.884185] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.894970] LustreError: Skipped 3 previous similar messages [ 365.048549] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.086394] LustreError: Skipped 5 previous similar messages [ 376.300984] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.332355] LustreError: Skipped 15 previous similar messages [ 380.988325] LDISKFS-fs (dm-0): recovery complete [ 380.993251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.678506] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2e137b to 0x908d15917c2e1a20 [ 392.678810] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 392.693809] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 392.710952] LustreError: Skipped 19 previous similar messages [ 392.821501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.339859] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 396.444873] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 398.310425] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 398.356815] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 403.918941] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 405.249882] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 414.314721] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 11:01:37 (1776178897) [ 417.465217] Lustre: Failing over lustre-OST0000 [ 417.514916] Lustre: server umount lustre-OST0000 complete [ 418.784764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 418.788471] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 418.814921] Lustre: Skipped 2 previous similar messages [ 425.062844] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 425.086629] LustreError: Skipped 5 previous similar messages [ 437.687327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 438.024278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.356736] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 439.717090] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 439.717224] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.204.130@tcp (at 0@lo) [ 439.730460] Lustre: Skipped 3 previous similar messages [ 439.740946] Lustre: lustre-OST0000: deleting orphan objects from 0x0:34 to 0x0:65 [ 443.207244] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 453.004891] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 454.708267] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 463.682909] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 11:02:26 (1776178946) [ 473.326109] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 475.983474] Lustre: Failing over lustre-MDT0000 [ 476.173247] Lustre: server umount lustre-MDT0000 complete [ 479.203537] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.225630] Lustre: Skipped 3 previous similar messages [ 485.217664] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776178964/real 1776178964] req@0000000087ad57f0 x1862458229739264/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776178971 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 485.238922] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 490.470987] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.478860] LustreError: Skipped 21 previous similar messages [ 500.181539] LDISKFS-fs (dm-0): recovery complete [ 500.183903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.757103] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2e1a20 to 0x908d15917c2e245a [ 502.781047] Lustre: MGC192.168.204.130@tcp: Connection restored to 192.168.204.130@tcp (at 0@lo) [ 502.787919] Lustre: Skipped 1 previous similar message [ 502.951237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.794498] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 508.384283] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 508.386870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 509.946811] Lustre: lustre-MDT0000: Denying connection for new client 32ab88b3-f92a-4cda-8d01-6c90b6a2944e (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 515.167686] Lustre: lustre-MDT0000: Denying connection for new client 32ab88b3-f92a-4cda-8d01-6c90b6a2944e (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 520.306244] Lustre: lustre-MDT0000: Denying connection for new client 32ab88b3-f92a-4cda-8d01-6c90b6a2944e (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 525.410449] Lustre: lustre-MDT0000: Denying connection for new client 32ab88b3-f92a-4cda-8d01-6c90b6a2944e (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 530.529681] Lustre: lustre-MDT0000: Denying connection for new client 32ab88b3-f92a-4cda-8d01-6c90b6a2944e (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:43 [ 540.768179] Lustre: lustre-MDT0000: Denying connection for new client 32ab88b3-f92a-4cda-8d01-6c90b6a2944e (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:33 [ 540.784457] Lustre: Skipped 1 previous similar message [ 561.252690] Lustre: lustre-MDT0000: Denying connection for new client 32ab88b3-f92a-4cda-8d01-6c90b6a2944e (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:12 [ 561.269845] Lustre: Skipped 3 previous similar messages [ 574.009864] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 574.021530] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 574.038158] Lustre: lustre-MDT0000: Recovery over after 1:06, of 2 clients 1 recovered and 1 was evicted. [ 574.038527] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 574.056936] Lustre: Skipped 2 previous similar messages [ 574.088890] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:65 [ 574.092723] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:97 [ 585.693720] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 11:04:28 (1776179068) [ 592.022962] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 593.702937] Lustre: Failing over lustre-MDT0000 [ 593.847564] Lustre: server umount lustre-MDT0000 complete [ 594.916346] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 594.929280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 594.944103] Lustre: Skipped 1 previous similar message [ 602.591777] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179081/real 1776179081] req@0000000025478efd x1862458229768064/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179088 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 602.621790] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.091509] LDISKFS-fs (dm-0): recovery complete [ 615.095910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.979546] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.995204] LustreError: Skipped 44 previous similar messages [ 620.003446] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2e245a to 0x908d15917c2e285f [ 620.013290] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 620.213825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 620.248290] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 623.716874] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 625.448417] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 625.455501] Lustre: lustre-MDT0000: Denying connection for new client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 625.471390] Lustre: Skipped 2 previous similar messages [ 691.000176] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 691.010949] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 691.031043] Lustre: lustre-MDT0000: Recovery over after 1:06, of 2 clients 1 recovered and 1 was evicted. [ 691.034694] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 691.073034] Lustre: Skipped 3 previous similar messages [ 691.102630] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:129 [ 691.103305] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:97 [ 700.080057] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 11:06:23 (1776179183) [ 707.241555] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 709.677747] Lustre: Failing over lustre-MDT0000 [ 710.097512] Lustre: server umount lustre-MDT0000 complete [ 711.657354] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 711.668610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.690087] Lustre: Skipped 3 previous similar messages [ 719.327420] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179198/real 1776179198] req@000000003080eb7b x1862458229795904/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179205 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 719.357581] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 731.679585] LDISKFS-fs (dm-0): recovery complete [ 731.691506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.748083] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2e285f to 0x908d15917c2e2ce2 [ 736.929077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.956988] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 737.376705] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 741.176822] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 742.476233] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 742.530365] Lustre: lustre-OST0000: deleting orphan objects from 0x0:76 to 0x0:161 [ 742.530810] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:129 [ 750.971327] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 752.321608] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 761.460967] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 11:07:24 (1776179244) [ 770.355642] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 772.870962] Lustre: Failing over lustre-MDT0000 [ 773.092880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.106977] Lustre: Skipped 4 previous similar messages [ 773.111484] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 773.164795] Lustre: server umount lustre-MDT0000 complete [ 779.231830] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179258/real 1776179258] req@000000006a8b292a x1862458229812800/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179265 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 779.255727] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 793.442233] LDISKFS-fs (dm-0): recovery complete [ 793.444581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 796.645554] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2e2ce2 to 0x908d15917c2e31dc [ 796.654901] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 796.657527] Lustre: Skipped 5 previous similar messages [ 796.873187] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 798.741767] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 800.691956] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 802.370606] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 802.429631] Lustre: lustre-OST0000: deleting orphan objects from 0x0:163 to 0x0:193 [ 802.430986] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:161 [ 808.928192] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 810.318610] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 817.766858] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 11:08:20 (1776179300) [ 824.305624] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 825.722346] Lustre: Failing over lustre-MDT0000 [ 825.813613] Lustre: server umount lustre-MDT0000 complete [ 827.877166] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.888467] Lustre: Skipped 3 previous similar messages [ 827.891943] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 835.039134] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179313/real 1776179313] req@000000003806de18 x1862458229829568/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179320 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 835.055394] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 848.163180] LDISKFS-fs (dm-0): recovery complete [ 848.173491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.426282] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2e31dc to 0x908d15917c2e3715 [ 851.699530] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 851.936688] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 857.188129] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 857.283077] Lustre: lustre-OST0000: deleting orphan objects from 0x0:195 to 0x0:225 [ 857.283396] Lustre: lustre-OST0001: deleting orphan objects from 0x0:44 to 0x0:193 [ 857.493876] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 870.092634] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 872.174720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 880.842154] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 11:09:24 (1776179364) [ 887.520800] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 889.206478] Lustre: Failing over lustre-MDT0000 [ 889.339932] Lustre: server umount lustre-MDT0000 complete [ 892.897179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.901357] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 892.907242] Lustre: Skipped 5 previous similar messages [ 892.918281] LustreError: Skipped 124 previous similar messages [ 899.935184] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179378/real 1776179378] req@0000000018e94a1b x1862458229848320/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179385 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 899.963587] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 911.484038] LDISKFS-fs (dm-0): recovery complete [ 911.489078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.451991] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2e3715 to 0x908d15917c2e3c32 [ 916.515919] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (not set up) [ 916.717386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.723875] Lustre: Skipped 2 previous similar messages [ 916.758195] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 917.935265] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 921.503321] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 922.253966] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 922.283286] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:257 [ 922.290143] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:225 [ 932.424184] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 934.186534] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 941.835805] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 11:10:24 (1776179424) [ 949.451179] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 959.967278] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179438/real 1776179438] req@00000000027a3f0f x1862458229866368/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179445 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 959.983915] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 973.985956] LDISKFS-fs (dm-0): recovery complete [ 973.989983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.353957] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2e3c32 to 0x908d15917c2e416b [ 976.367412] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 976.378990] Lustre: Skipped 14 previous similar messages [ 976.641057] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 981.946854] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 982.143448] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:289 [ 982.143816] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:257 [ 991.849581] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 994.036209] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1002.130996] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 11:11:25 (1776179485) [ 1003.180215] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1003.186436] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1003.193735] LustreError: 6265:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000e22ec452 x1862458208754304/t38654705666(0) o35->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:469/0 lens 392/456 e 0 to 0 dl 1776179494 ref 1 fl Interpret:/0/0 rc 0/0 job:'openfile.0' [ 1010.181637] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1010.326443] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 1010.340401] Lustre: 6265:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000b2efffba x1862458208754304/t38654705666(0) o35->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:477/0 lens 392/456 e 0 to 0 dl 1776179502 ref 1 fl Interpret:/2/0 rc 0/0 job:'openfile.0' [ 1011.474647] Lustre: Failing over lustre-MDT0000 [ 1011.477969] Lustre: Skipped 1 previous similar message [ 1011.599125] Lustre: server umount lustre-MDT0000 complete [ 1011.602995] Lustre: Skipped 1 previous similar message [ 1012.703469] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1012.707526] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1012.728309] Lustre: Skipped 6 previous similar messages [ 1032.519939] LDISKFS-fs (dm-0): recovery complete [ 1032.527473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.990415] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1037.920719] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1037.931125] Lustre: Skipped 1 previous similar message [ 1040.825135] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1042.450083] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1042.455134] Lustre: Skipped 1 previous similar message [ 1042.496310] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:289 [ 1042.499890] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:321 [ 1049.948485] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1051.352557] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1059.324637] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 11:12:22 (1776179542) [ 1066.510170] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1068.652183] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 1079.263443] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179558/real 1776179558] req@00000000cb81e138 x1862458229901888/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179565 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 1079.295109] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1079.303437] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.312950] LustreError: Skipped 1 previous similar message [ 1093.195923] LDISKFS-fs (dm-0): recovery complete [ 1093.199965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.674837] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2e4649 to 0x908d15917c2e4afd [ 1096.683493] Lustre: Skipped 1 previous similar message [ 1096.921575] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1101.282875] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1102.437682] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:353 [ 1102.440636] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:321 [ 1110.873251] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1112.602627] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1120.453166] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 11:13:23 (1776179603) [ 1128.424963] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1129.363901] Lustre: *** cfs_fail_loc=114, val=0*** [ 1155.108413] LDISKFS-fs (dm-0): recovery complete [ 1155.111856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.707656] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (not set up) [ 1161.277779] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1162.319359] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:385 [ 1162.319724] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:353 [ 1170.349631] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1171.809264] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1179.519650] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 11:14:22 (1776179662) [ 1185.089465] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1185.841189] Lustre: *** cfs_fail_loc=128, val=0*** [ 1187.588901] Lustre: Failing over lustre-MDT0000 [ 1187.590826] Lustre: Skipped 2 previous similar messages [ 1187.748624] Lustre: server umount lustre-MDT0000 complete [ 1187.751602] Lustre: Skipped 2 previous similar messages [ 1187.813337] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.814798] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1187.825614] Lustre: Skipped 12 previous similar messages [ 1211.901177] LDISKFS-fs (dm-0): recovery complete [ 1211.912700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.723799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1216.726971] Lustre: Skipped 4 previous similar messages [ 1216.829570] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1216.844361] Lustre: Skipped 1 previous similar message [ 1217.128310] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1217.138855] Lustre: Skipped 2 previous similar messages [ 1221.339472] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1222.176866] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1222.180262] Lustre: Skipped 2 previous similar messages [ 1222.257483] Lustre: lustre-OST0000: deleting orphan objects from 0x0:227 to 0x0:417 [ 1222.257881] Lustre: lustre-OST0001: deleting orphan objects from 0x0:195 to 0x0:385 [ 1231.973666] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1233.553281] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1242.561571] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 11:15:25 (1776179725) [ 1252.896889] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1257.951949] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1265.119355] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776179743/real 1776179743] req@000000000411c283 x1862458229954048/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776179750 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 1265.165275] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1265.184289] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1265.200186] LustreError: Skipped 2 previous similar messages [ 1276.968053] LDISKFS-fs (dm-0): recovery complete [ 1276.971253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.532538] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2e5496 to 0x908d15917c2e5abd [ 1282.545225] Lustre: Skipped 2 previous similar messages [ 1282.549216] LustreError: 39758:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.204.130@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (0000000030302a00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 1282.565668] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 1282.571410] Lustre: Skipped 24 previous similar messages [ 1282.682774] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (not set up) [ 1286.638403] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1288.396180] Lustre: lustre-OST0001: deleting orphan objects from 0x0:391 to 0x0:417 [ 1288.396816] Lustre: lustre-OST0000: deleting orphan objects from 0x0:423 to 0x0:449 [ 1294.897158] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1296.391636] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1305.761669] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 11:16:28 (1776179788) [ 1314.773232] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1338.845098] LDISKFS-fs (dm-0): recovery complete [ 1338.872929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.097277] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (not set up) [ 1347.871505] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1349.710952] Lustre: lustre-OST0001: deleting orphan objects from 0x0:423 to 0x0:449 [ 1349.711276] Lustre: lustre-OST0000: deleting orphan objects from 0x0:455 to 0x0:481 [ 1355.773832] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1357.196613] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1364.289909] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 11:17:27 (1776179847) [ 1370.537229] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1396.429860] LDISKFS-fs (dm-0): recovery complete [ 1396.432516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.610456] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1404.617307] Lustre: Skipped 2 previous similar messages [ 1408.260053] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1413.368771] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:577 [ 1413.371335] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:609 [ 1419.308368] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1421.163232] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1446.563428] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 11:18:49 (1776179929) [ 1453.879525] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1455.677786] Lustre: Failing over lustre-MDT0000 [ 1455.679541] Lustre: Skipped 3 previous similar messages [ 1455.885210] Lustre: server umount lustre-MDT0000 complete [ 1455.893415] Lustre: Skipped 3 previous similar messages [ 1456.106808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1456.110655] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.131901] Lustre: Skipped 14 previous similar messages [ 1456.165165] LustreError: Skipped 330 previous similar messages [ 1478.113621] LDISKFS-fs (dm-0): recovery complete [ 1478.116929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.288727] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1480.294024] Lustre: Skipped 3 previous similar messages [ 1484.230240] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1485.411131] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1485.419840] Lustre: Skipped 3 previous similar messages [ 1485.477986] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:641 [ 1485.479460] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:609 [ 1493.964202] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1496.157417] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1507.293132] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 11:19:50 (1776179990) [ 1515.857120] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1527.263322] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180006/real 1776180006] req@00000000dec9f422 x1862458230064000/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180013 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 1527.292706] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1527.298276] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1527.313185] LustreError: Skipped 3 previous similar messages [ 1539.591265] LDISKFS-fs (dm-0): recovery complete [ 1539.598016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.681278] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2ef343 to 0x908d15917c2ef86e [ 1544.692627] Lustre: Skipped 3 previous similar messages [ 1544.809401] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (not set up) [ 1549.635393] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1550.394872] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:641 [ 1550.395423] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:673 [ 1559.806688] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1561.171419] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1569.677100] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 11:20:52 (1776180052) [ 1577.451211] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1581.026120] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1599.831844] LDISKFS-fs (dm-0): recovery complete [ 1599.836803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.632418] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1610.438975] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:673 [ 1610.444564] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:705 [ 1621.052657] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1623.375584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1633.815512] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 11:21:56 (1776180116) [ 1641.467725] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1667.516921] LDISKFS-fs (dm-0): recovery complete [ 1667.535025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.958509] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1670.974712] Lustre: Skipped 3 previous similar messages [ 1675.618446] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1676.486484] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:705 [ 1676.486592] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:737 [ 1685.557930] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1687.618925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1696.934277] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 11:23:00 (1776180180) [ 1703.594712] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1725.678843] LDISKFS-fs (dm-0): recovery complete [ 1725.685640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.739979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.745359] Lustre: Skipped 7 previous similar messages [ 1735.389191] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1736.300489] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:769 [ 1736.301289] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:737 [ 1745.734937] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1747.890528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1757.588287] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 11:24:00 (1776180240) [ 1764.877563] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1766.885709] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1789.490974] LDISKFS-fs (dm-0): recovery complete [ 1789.497783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.567285] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 1795.576539] Lustre: Skipped 39 previous similar messages [ 1795.681981] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (not set up) [ 1799.742686] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1801.347967] Lustre: lustre-OST0000: deleting orphan objects from 0x0:592 to 0x0:801 [ 1801.350114] Lustre: lustre-OST0001: deleting orphan objects from 0x0:560 to 0x0:769 [ 1809.489963] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1811.112919] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1820.221304] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 11:25:03 (1776180303) [ 1826.632123] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1849.348355] LDISKFS-fs (dm-0): recovery complete [ 1849.352840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.134451] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1861.806902] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:833 [ 1861.810450] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:801 [ 1871.288221] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1874.606535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1884.529842] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 11:26:07 (1776180367) [ 1892.250859] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1914.633092] LDISKFS-fs (dm-0): recovery complete [ 1914.635621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.912816] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1927.747180] Lustre: lustre-OST0001: deleting orphan objects from 0x0:771 to 0x0:833 [ 1927.748123] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:865 [ 1933.962707] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1935.321796] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1943.340722] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 11:27:06 (1776180426) [ 1950.520336] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1972.289857] LDISKFS-fs (dm-0): recovery complete [ 1972.294783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.770075] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 1983.577360] Lustre: lustre-OST0000: deleting orphan objects from 0x0:803 to 0x0:897 [ 1983.578237] Lustre: lustre-OST0001: deleting orphan objects from 0x0:835 to 0x0:865 [ 1991.985455] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1993.654701] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2003.261935] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 11:28:05 (1776180485) [ 2011.557658] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2013.414862] Lustre: Failing over lustre-MDT0000 [ 2013.418246] Lustre: Skipped 8 previous similar messages [ 2013.548340] Lustre: server umount lustre-MDT0000 complete [ 2013.558566] Lustre: Skipped 8 previous similar messages [ 2014.177086] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.192010] Lustre: Skipped 35 previous similar messages [ 2036.346564] LDISKFS-fs (dm-0): recovery complete [ 2036.350389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.369132] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2038.378597] Lustre: Skipped 8 previous similar messages [ 2042.998265] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2043.437234] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2043.442846] Lustre: Skipped 8 previous similar messages [ 2043.468277] Lustre: lustre-OST0001: deleting orphan objects from 0x0:835 to 0x0:897 [ 2043.472748] Lustre: lustre-OST0000: deleting orphan objects from 0x0:899 to 0x0:929 [ 2052.540507] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2054.167441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2062.281547] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 11:29:05 (1776180545) [ 2068.779399] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2074.080698] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2074.081551] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.099300] LustreError: Skipped 3 previous similar messages [ 2074.116860] LustreError: Skipped 378 previous similar messages [ 2080.223311] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776180559/real 1776180559] req@0000000024b5d185 x1862458230225792/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776180566 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 2080.236081] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2080.241901] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2080.248261] LustreError: Skipped 8 previous similar messages [ 2092.745748] LDISKFS-fs (dm-0): recovery complete [ 2092.758193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2097.632581] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2f23b0 to 0x908d15917c2f28f7 [ 2097.651213] Lustre: Skipped 8 previous similar messages [ 2097.760837] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (not set up) [ 2101.727325] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2103.422908] Lustre: lustre-OST0001: deleting orphan objects from 0x0:899 to 0x0:929 [ 2103.423915] Lustre: lustre-OST0000: deleting orphan objects from 0x0:931 to 0x0:961 [ 2109.545042] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2111.076085] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2119.505144] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 11:30:02 (1776180602) [ 2127.537922] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2151.727090] LDISKFS-fs (dm-0): recovery complete [ 2151.734203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.443303] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2163.883787] Lustre: lustre-OST0001: deleting orphan objects from 0x0:899 to 0x0:961 [ 2163.888630] Lustre: lustre-OST0000: deleting orphan objects from 0x0:963 to 0x0:993 [ 2172.965263] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2175.114831] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2184.843607] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 11:31:07 (1776180667) [ 2192.174824] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2194.404249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.415011] Lustre: Skipped 3 previous similar messages [ 2216.240357] LDISKFS-fs (dm-0): recovery complete [ 2216.245724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2224.450661] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2224.458750] Lustre: Skipped 8 previous similar messages [ 2229.079675] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2229.924511] Lustre: lustre-OST0000: deleting orphan objects from 0x0:995 to 0x0:1025 [ 2229.929925] Lustre: lustre-OST0001: deleting orphan objects from 0x0:899 to 0x0:993 [ 2239.688571] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2241.517409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2250.089512] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 11:32:13 (1776180733) [ 2256.910414] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2280.210043] LDISKFS-fs (dm-0): recovery complete [ 2280.213230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.701333] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2290.777887] Lustre: lustre-OST0001: deleting orphan objects from 0x0:995 to 0x0:1025 [ 2290.778967] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2297.277063] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2298.934562] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2307.400639] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 11:33:10 (1776180790) [ 2313.465871] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2334.625282] LDISKFS-fs (dm-0): recovery complete [ 2334.628555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.029305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2341.041349] Lustre: Skipped 9 previous similar messages [ 2344.640921] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2346.545566] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1059 to 0x0:1089 [ 2346.545618] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1057 [ 2353.626917] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2355.187436] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2362.908039] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:34:06 (1776180846) [ 2369.424673] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2393.205588] LDISKFS-fs (dm-0): recovery complete [ 2393.211761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.646279] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 2396.653700] Lustre: Skipped 49 previous similar messages [ 2400.702492] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2402.353140] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1089 [ 2402.358950] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1091 to 0x0:1121 [ 2407.684856] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2408.803509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2414.680922] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 11:34:58 (1776180898) [ 2417.434442] Lustre: 77575:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 543b08dc-b2ae-46cd-aa25-e10cda298cc8 at adminstrative request [ 2439.940903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2444.687835] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2446.904690] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 2446.906399] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1027 to 0x0:1121 [ 2453.138182] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2454.536215] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2459.672453] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2466.897078] Lustre: DEBUG MARKER: before 2844, after 2844 [ 2473.658544] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 11:35:56 (1776180956) [ 2474.640247] Lustre: 79608:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 543b08dc-b2ae-46cd-aa25-e10cda298cc8 at adminstrative request [ 2485.218354] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:36:08 (1776180968) [ 2493.118140] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2519.430547] LDISKFS-fs (dm-0): recovery complete [ 2519.433188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.352814] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2527.414466] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1123 to 0x0:1153 [ 2527.416972] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1156 to 0x0:1185 [ 2533.878683] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2535.086955] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2541.447555] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:37:04 (1776181024) [ 2547.554989] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2571.439209] LDISKFS-fs (dm-0): recovery complete [ 2571.442955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.285643] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2583.645794] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2583.649990] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1155 to 0x0:1185 [ 2592.334530] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2593.541658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2599.637347] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 11:38:03 (1776181083) [ 2606.532937] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2609.120729] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2609.131257] LustreError: Skipped 5 previous similar messages [ 2630.790430] LDISKFS-fs (dm-0): recovery complete [ 2630.793564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2637.657341] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2638.372092] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1187 to 0x0:1217 [ 2638.373262] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2646.356303] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2647.982203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2655.110840] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 11:38:58 (1776181138) [ 2662.132421] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2663.904167] Lustre: Failing over lustre-MDT0000 [ 2663.908061] Lustre: Skipped 10 previous similar messages [ 2663.911445] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.916728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.930909] Lustre: Skipped 43 previous similar messages [ 2663.939300] Lustre: Skipped 3 previous similar messages [ 2664.040498] Lustre: server umount lustre-MDT0000 complete [ 2664.048241] Lustre: Skipped 10 previous similar messages [ 2674.144193] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.164292] LustreError: Skipped 379 previous similar messages [ 2687.906987] LDISKFS-fs (dm-0): recovery complete [ 2687.911286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.417892] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2694.430566] Lustre: Skipped 10 previous similar messages [ 2698.304354] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2698.315670] Lustre: Skipped 10 previous similar messages [ 2698.398204] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1219 to 0x0:1249 [ 2698.399583] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2699.023733] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2708.682511] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2710.548443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2719.282512] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 11:40:02 (1776181202) [ 2727.084470] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2740.703210] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181219/real 1776181219] req@00000000f6c1fac9 x1862458230419648/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181226 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 2740.723864] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 2740.730184] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2740.740803] LustreError: Skipped 10 previous similar messages [ 2750.951061] LDISKFS-fs (dm-0): recovery complete [ 2750.954921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.114504] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2f5fa0 to 0x908d15917c2f64fc [ 2758.130768] Lustre: Skipped 10 previous similar messages [ 2762.356602] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2763.858504] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1251 to 0x0:1313 [ 2763.861372] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1251 to 0x0:1281 [ 2770.713269] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2772.744459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2780.701764] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:41:03 (1776181263) [ 2787.636604] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2812.024957] LDISKFS-fs (dm-0): recovery complete [ 2812.028804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.546354] LustreError: 91179:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.204.130@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000a7ca082c) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2823.187462] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2824.314473] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1283 to 0x0:1313 [ 2824.321716] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2831.545181] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2832.957325] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2840.058755] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:42:03 (1776181323) [ 2846.219383] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2871.272206] LDISKFS-fs (dm-0): recovery complete [ 2871.277140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.100477] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2875.103506] Lustre: Skipped 10 previous similar messages [ 2878.997465] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2880.614579] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2880.619428] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1315 to 0x0:1345 [ 2888.200394] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2889.853384] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2896.886291] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 11:43:00 (1776181380) [ 2903.800795] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2927.818589] LDISKFS-fs (dm-0): recovery complete [ 2927.824540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2933.750121] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 2935.382804] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 2935.387115] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1347 to 0x0:1377 [ 2945.099341] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2947.064217] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2956.497857] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:43:59 (1776181439) [ 2963.906917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2965.994063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2966.003436] Lustre: Skipped 4 previous similar messages [ 2988.155704] LDISKFS-fs (dm-0): recovery complete [ 2988.159438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2994.883764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.890148] Lustre: Skipped 10 previous similar messages [ 2998.815945] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3000.306136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 3000.310022] Lustre: Skipped 50 previous similar messages [ 3000.416688] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1379 to 0x0:1409 [ 3000.419725] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 3008.162663] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3009.701820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3017.278943] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 11:45:00 (1776181500) [ 3024.007520] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3045.495456] LDISKFS-fs (dm-0): recovery complete [ 3045.499034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.137843] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3060.306562] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 3060.307621] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1411 to 0x0:1441 [ 3067.673472] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3069.041926] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3076.477911] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 11:45:59 (1776181559) [ 3082.912691] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3107.118791] LDISKFS-fs (dm-0): recovery complete [ 3107.133426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.084252] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1443 to 0x0:1473 [ 3115.085208] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1475 to 0x0:1505 [ 3115.338839] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3128.678500] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3131.709368] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3141.633896] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 11:47:04 (1776181624) [ 3142.736130] Lustre: 102302:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 543b08dc-b2ae-46cd-aa25-e10cda298cc8 at adminstrative request [ 3153.454625] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 11:47:16 (1776181636) [ 3159.256530] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3172.979749] LDISKFS-fs (dm-0): recovery complete [ 3172.985788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.633960] LustreError: 103584:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3174.640305] LustreError: 103584:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3174.646772] Lustre: 103618:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3174.653983] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3174.660460] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3174.674685] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3174.718952] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1512 to 0x0:1537 [ 3174.719819] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1479 to 0x0:1505 [ 3178.320299] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3180.007101] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3196.842954] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 11:48:00 (1776181680) [ 3202.882226] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3214.932159] LDISKFS-fs (dm-0): recovery complete [ 3214.935514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.055748] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3219.068422] LustreError: 105500:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3219.073186] LustreError: 105500:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3219.081350] Lustre: 105534:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3219.086638] Lustre: 105534:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3219.092908] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3219.098941] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3219.112124] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3219.158243] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1511 to 0x0:1537 [ 3219.159837] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1543 to 0x0:1569 [ 3223.089335] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3224.552344] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3235.209588] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3235.212753] Lustre: Skipped 1 previous similar message [ 3239.292732] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 11:48:42 (1776181722) [ 3245.382939] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3250.149736] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3250.154342] LustreError: Skipped 9 previous similar messages [ 3258.061487] LDISKFS-fs (dm-0): recovery complete [ 3258.064450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3263.163973] LustreError: 107418:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3263.168905] LustreError: 107418:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3263.173817] Lustre: 107452:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3263.178724] Lustre: 107452:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3263.182964] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3263.189038] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3263.199919] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3263.239475] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1601 [ 3263.240224] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1544 to 0x0:1569 [ 3266.780201] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3268.580398] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3284.104601] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 11:49:27 (1776181767) [ 3285.087113] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3285.094053] LustreError: 6261:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000359fc1f8 x1862458209294912/t201863462916(0) o36->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:521/0 lens 504/456 e 0 to 0 dl 1776181811 ref 1 fl Interpret:/0/0 rc 0/0 job:'rm.0' [ 3288.738077] Lustre: Failing over lustre-MDT0000 [ 3288.740967] Lustre: Skipped 10 previous similar messages [ 3288.912073] Lustre: server umount lustre-MDT0000 complete [ 3288.917822] Lustre: Skipped 10 previous similar messages [ 3289.060718] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.064303] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.078100] Lustre: Skipped 43 previous similar messages [ 3289.092953] LustreError: Skipped 366 previous similar messages [ 3296.074968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3296.379179] LustreError: 108797:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3296.384316] LustreError: 108797:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3296.388417] Lustre: 108830:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3296.393852] Lustre: 108830:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3296.398531] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3296.405852] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3296.427834] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3296.480577] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1571 to 0x0:1601 [ 3296.509381] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1575 to 0x0:1633 [ 3301.225823] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3301.361066] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3321.118081] Lustre: DEBUG MARKER: == replay-single test 36: don't resend cancel ============ 11:50:04 (1776181804) [ 3328.308676] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3349.574993] LDISKFS-fs (dm-0): recovery complete [ 3349.577542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.319662] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3358.328828] Lustre: Skipped 7 previous similar messages [ 3360.177179] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3361.877424] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3361.887995] Lustre: Skipped 7 previous similar messages [ 3361.927761] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1571 to 0x0:1633 [ 3361.931055] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1665 [ 3369.332348] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 11:50:52 (1776181852) [ 3379.052611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3394.892460] LDISKFS-fs (dm-0): recovery complete [ 3394.895456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.980025] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3394.989722] LustreError: Skipped 11 previous similar messages [ 3394.994581] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c2faa70 to 0x908d15917c2fafcc [ 3394.997474] Lustre: Skipped 11 previous similar messages [ 3395.144979] LustreError: 112561:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3395.155315] LustreError: 112561:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3395.161453] Lustre: 112594:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3395.171491] Lustre: 112594:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3395.176411] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3395.188887] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3395.221426] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3395.274918] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1697 [ 3395.274992] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1571 to 0x0:1665 [ 3398.488253] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3400.174745] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3417.032976] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:51:40 (1776181900) [ 3446.529658] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3458.528084] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776181937/real 1776181937] req@00000000656dc49a x1862458230673216/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776181944 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 3458.569149] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [ 3471.149810] LDISKFS-fs (dm-0): recovery complete [ 3471.152863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3476.189283] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3476.203637] Lustre: Skipped 20 previous similar messages [ 3480.351206] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3481.732269] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2098 to 0x0:2113 [ 3489.047899] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3490.641154] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3507.271810] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 11:53:10 (1776181990) [ 3528.445103] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3537.897662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3537.901949] Lustre: Skipped 1 previous similar message [ 3562.409335] LDISKFS-fs (dm-0): recovery complete [ 3562.412295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.483984] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3575.547038] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2914 to 0x0:2945 [ 3575.547221] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2066 to 0x0:2081 [ 3580.164436] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3581.289176] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3598.989367] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 11:54:42 (1776182082) [ 3600.678459] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO [ 3602.092475] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 11:54:45 (1776182085) [ 3603.692891] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3604.495715] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3604.501996] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3604.511154] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to (at 0@lo) [ 3604.514461] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2066 to 0x0:2113 [ 3604.516088] Lustre: Skipped 53 previous similar messages [ 3609.780666] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 11:54:52 (1776182092) [ 3635.688433] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3668.394437] LDISKFS-fs (dm-2): recovery complete [ 3668.398313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3668.568582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.571938] Lustre: Skipped 10 previous similar messages [ 3670.848929] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65 [ 3670.857453] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3347 to 0x0:3393 [ 3672.205331] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3728.525452] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 11:56:51 (1776182211) [ 3738.133065] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3763.029509] LDISKFS-fs (dm-0): recovery complete [ 3763.031457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3774.555703] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3775.559144] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3775.560991] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2514 to 0x0:2529 [ 3782.375591] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3782.624717] LustreError: 122408:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3782.626138] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3783.580960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3783.648534] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3347 to 0x0:3425 [ 3800.645673] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 11:58:03 (1776182283) [ 3804.125746] LustreError: 17936:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3809.247440] LustreError: 17936:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3809.252078] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 3809.287057] LustreError: 44562:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3810.414445] LustreError: 17936:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3815.526945] LustreError: 11236:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3815.532151] LustreError: 11236:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message [ 3815.536466] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 3815.537226] LustreError: 17936:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=1 [ 3816.865647] LustreError: 17936:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3822.047156] LustreError: 17936:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3822.054566] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 3822.062496] Lustre: Skipped 1 previous similar message [ 3823.554818] LustreError: 11236:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3828.708573] LustreError: 11236:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3830.303150] LustreError: 6263:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3835.359427] LustreError: 6263:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3835.369212] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 3835.382087] Lustre: Skipped 1 previous similar message [ 3843.580777] LustreError: 17936:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3843.585687] LustreError: 17936:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message [ 3848.671133] LustreError: 17936:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3848.681394] LustreError: 17936:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message [ 3855.839341] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 3855.852768] Lustre: Skipped 2 previous similar messages [ 3862.499477] LustreError: 6261:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking [ 3864.015528] LustreError: 6261:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping [ 3864.018548] LustreError: 6261:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages [ 3869.151137] LustreError: 6261:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0 [ 3869.157292] LustreError: 6261:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages [ 3877.157518] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 11:59:20 (1776182360) [ 3878.851530] LustreError: 11236:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3885.165037] Lustre: lustre-MDT0000: Export 0000000086a3f550 already connecting from 192.168.204.30@tcp [ 3886.709992] Lustre: lustre-MDT0000: Export 0000000086a3f550 already connecting from 192.168.204.30@tcp [ 3888.122352] Lustre: lustre-MDT0000: Export 0000000086a3f550 already connecting from 192.168.204.30@tcp [ 3891.424441] Lustre: lustre-MDT0000: Export 0000000086a3f550 already connecting from 192.168.204.30@tcp [ 3891.429116] Lustre: Skipped 1 previous similar message [ 3895.673427] Lustre: lustre-MDT0000: Export 0000000086a3f550 already connecting from 192.168.204.30@tcp [ 3895.678866] Lustre: Skipped 3 previous similar messages [ 3900.087415] LustreError: 11236:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 3900.094618] Lustre: 11236:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/1s); client may timeout req@0000000014231264 x1862458210324352/t0(0) o38->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:0/0 lens 520/416 e 0 to 0 dl 1776182384 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0' [ 3900.510398] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 3900.519509] Lustre: Skipped 4 previous similar messages [ 3902.974960] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 11:59:46 (1776182386) [ 3907.779781] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3909.831518] Lustre: Failing over lustre-MDT0000 [ 3909.835235] Lustre: Skipped 6 previous similar messages [ 3909.957291] Lustre: server umount lustre-MDT0000 complete [ 3909.959897] Lustre: Skipped 6 previous similar messages [ 3910.625171] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.625595] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.631402] Lustre: Skipped 27 previous similar messages [ 3910.641545] LustreError: Skipped 191 previous similar messages [ 3916.209493] LDISKFS-fs (dm-0): recovery complete [ 3916.212867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.299773] LustreError: 11-0: MGC192.168.204.130@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 3916.305764] LustreError: Skipped 7 previous similar messages [ 3916.310119] LustreError: 127090:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.204.130@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000a4f0b14e) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3916.396476] Lustre: *** cfs_fail_loc=712, val=0*** [ 3916.399811] LustreError: 8237:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery req@00000000ed71296f x1862458230959296/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' [ 3916.411298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3916.448621] LustreError: 127081:0:(mdt_handler.c:7436:mdt_iocontrol()) lustre-MDT0000: Aborting client recovery [ 3916.452092] LustreError: 127081:0:(ldlm_lib.c:2902:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3916.456338] Lustre: 127115:0:(ldlm_lib.c:2290:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3916.463243] Lustre: 127115:0:(ldlm_lib.c:2290:target_recovery_overseer()) Skipped 2 previous similar messages [ 3916.469084] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3916.475631] Lustre: lustre-MDT0000-osd: cancel update llog [0x200018aa0:0x1:0x0] [ 3916.490310] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3916.521667] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3347 to 0x0:3457 [ 3916.521975] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2514 to 0x0:2561 [ 3918.955243] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3921.897415] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3950.609936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.470820] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3963.485221] Lustre: Skipped 4 previous similar messages [ 3967.058935] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 3967.500692] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3967.514558] Lustre: Skipped 4 previous similar messages [ 3967.575516] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2514 to 0x0:2593 [ 3967.576076] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3347 to 0x0:3489 [ 3980.534915] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3982.912545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3993.048706] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:01:15 (1776182475) [ 3993.153344] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 3999.739567] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:01:22 (1776182482) [ 4000.854276] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4000.861989] LustreError: 6271:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000066cb18f x1862458210350016/t0(0) o700->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:447/0 lens 264/248 e 0 to 0 dl 1776182492 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0' [ 4020.704042] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4020.743703] LustreError: Skipped 5 previous similar messages [ 4030.450656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4038.119236] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c32dca5 to 0x908d15917c32e13d [ 4038.126348] Lustre: Skipped 5 previous similar messages [ 4042.741734] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4043.849290] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2595 to 0x0:2625 [ 4043.852080] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521 [ 4050.673508] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4051.911351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4059.408242] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:02:22 (1776182542) [ 4079.329777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4079.534271] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4079.543357] Lustre: Skipped 8 previous similar messages [ 4080.993134] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97 [ 4081.000070] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3532 to 0x0:3553 [ 4083.665981] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4094.426293] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4096.722632] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4169.739983] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:04:12 (1776182652) [ 4176.833894] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4189.151904] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776182667/real 1776182667] req@00000000c720111f x1862458231034176/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776182674 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 4189.174736] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 4202.912213] LDISKFS-fs (dm-0): recovery complete [ 4202.915598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4205.544305] Lustre: MGC192.168.204.130@tcp: Connection restored to 192.168.204.130@tcp (at 0@lo) [ 4205.560192] Lustre: Skipped 26 previous similar messages [ 4211.688160] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4211.897366] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2646 to 0x0:2689 [ 4211.901671] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3564 to 0x0:3585 [ 4286.799535] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:06:09 (1776182769) [ 4288.748667] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4288.755376] Lustre: Skipped 2 previous similar messages [ 4288.764812] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3617 [ 4289.589209] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3649 [ 4301.831432] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:06:24 (1776182784) [ 4327.060181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4333.354669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4333.371945] Lustre: Skipped 6 previous similar messages [ 4338.554724] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4338.682333] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4338.686607] LustreError: 135239:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000e0f7936a x1862458210408064/t0(0) o101->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:60/0 lens 328/344 e 0 to 0 dl 1776182860 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0' [ 4378.207460] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 4378.294523] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2701 to 0x0:2721 [ 4378.296605] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3681 [ 4385.959959] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4387.823375] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4398.964701] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:08:01 (1776182881) [ 4400.945171] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4409.609240] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4436.976333] LDISKFS-fs (dm-0): recovery complete [ 4436.982979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.814545] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4445.239746] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3713 [ 4445.240477] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2723 to 0x0:2753 [ 4454.714835] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4456.885682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4465.627840] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:09:08 (1776182948) [ 4466.805396] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4476.174106] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4501.570909] LDISKFS-fs (dm-0): recovery complete [ 4501.575527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.988274] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4510.235193] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2755 to 0x0:2785 [ 4510.240177] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3745 [ 4518.644460] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4520.032886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4526.024933] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:10:09 (1776183009) [ 4526.723991] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4533.009413] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4534.901259] Lustre: Failing over lustre-MDT0000 [ 4534.908670] Lustre: Skipped 7 previous similar messages [ 4535.089250] Lustre: server umount lustre-MDT0000 complete [ 4535.092695] Lustre: Skipped 7 previous similar messages [ 4535.399092] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4535.410613] LustreError: Skipped 246 previous similar messages [ 4535.782257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.789382] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4535.808302] Lustre: Skipped 31 previous similar messages [ 4535.824918] LustreError: Skipped 4 previous similar messages [ 4561.323438] LDISKFS-fs (dm-0): recovery complete [ 4561.329443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4577.062522] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4577.076191] Lustre: Skipped 6 previous similar messages [ 4581.001973] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4581.402316] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4581.413871] Lustre: Skipped 6 previous similar messages [ 4581.467641] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3777 [ 4581.469965] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2787 to 0x0:2817 [ 4593.317820] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 12:11:16 (1776183076) [ 4595.320770] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4595.325865] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4595.332105] LustreError: 6264:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000a3125398 x1862458210440768/t261993005073(0) o35->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:287/0 lens 392/456 e 0 to 0 dl 1776183087 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4597.864956] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 4597.870725] Lustre: Skipped 5 previous similar messages [ 4617.438158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4630.810258] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4631.590737] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3596 to 0x0:3809 [ 4631.599263] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2819 to 0x0:2849 [ 4631.630481] Lustre: 6264:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000009acb120 x1862458210440768/t261993005073(0) o35->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:323/0 lens 392/456 e 0 to 0 dl 1776183123 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4640.810696] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4642.406519] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4650.584166] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 12:12:13 (1776183133) [ 4651.650839] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4651.659460] LustreError: 6261:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000666e890f x1862458210449152/t266287972368(0) o36->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:377/0 lens 504/448 e 0 to 0 dl 1776183177 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4660.889628] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4674.464312] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4674.479455] LustreError: Skipped 6 previous similar messages [ 4684.199461] LDISKFS-fs (dm-0): recovery complete [ 4684.202637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.941654] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c331ce0 to 0x908d15917c332354 [ 4691.956971] Lustre: Skipped 6 previous similar messages [ 4692.202094] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4692.207146] Lustre: Skipped 6 previous similar messages [ 4696.401982] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4697.633760] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2819 to 0x0:2881 [ 4697.638726] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3811 to 0x0:3841 [ 4697.649051] Lustre: 9311:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000036098487 x1862458210449152/t266287972368(0) o36->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:423/0 lens 504/448 e 0 to 0 dl 1776183223 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4706.687362] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4708.252641] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4716.453233] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 12:13:19 (1776183199) [ 4717.548647] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4717.554226] LustreError: 17936:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000000541a8aa x1862458210458496/t270582939664(0) o36->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:443/0 lens 504/448 e 0 to 0 dl 1776183243 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4719.315106] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4726.686226] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4727.422278] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 4727.431578] Lustre: Skipped 2 previous similar messages [ 4727.449570] Lustre: 6263:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000057b53614 x1862458210458496/t270582939664(0) o36->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:453/0 lens 504/448 e 0 to 0 dl 1776183253 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4727.480312] Lustre: 6263:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4749.447233] LDISKFS-fs (dm-0): recovery complete [ 4749.451461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.382836] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4763.660467] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2883 to 0x0:2913 [ 4763.664238] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3811 to 0x0:3873 [ 4772.996382] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 12:14:15 (1776183255) [ 4774.209415] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4774.213478] Lustre: Skipped 1 previous similar message [ 4774.220265] LustreError: 9311:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000033a4a26f x1862458210467136/t274877906960(0) o36->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:499/0 lens 504/448 e 0 to 0 dl 1776183299 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0' [ 4774.237991] LustreError: 9311:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 4776.243232] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4776.247921] Lustre: Skipped 1 previous similar message [ 4782.708700] Lustre: 6262:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000009cb8d12c x1862458210467136/t274877906960(0) o36->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:508/0 lens 504/448 e 0 to 0 dl 1776183308 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0' [ 4784.320889] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4796.383266] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183275/real 1776183275] req@0000000094930606 x1862458231205184/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183282 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 4796.407115] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 4806.873684] LDISKFS-fs (dm-0): recovery complete [ 4806.877446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4813.797984] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 4813.803026] Lustre: Skipped 41 previous similar messages [ 4817.932237] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4819.560171] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2883 to 0x0:2945 [ 4819.561187] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3875 to 0x0:3905 [ 4827.893619] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 12:15:11 (1776183311) [ 4828.753448] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4830.460547] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4830.464873] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4830.471345] LustreError: 6264:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000006599b5fc x1862458210475648/t279172874256(0) o35->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:522/0 lens 392/456 e 0 to 0 dl 1776183322 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0' [ 4836.967066] Lustre: 6264:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000006b65f2ca x1862458210475648/t279172874256(0) o35->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:528/0 lens 392/456 e 0 to 0 dl 1776183328 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0' [ 4837.649976] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4861.300234] LDISKFS-fs (dm-0): recovery complete [ 4861.302824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4873.153478] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4874.351948] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2947 to 0x0:2977 [ 4874.362412] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3875 to 0x0:3937 [ 4883.560316] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 12:16:07 (1776183367) [ 4884.342131] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4884.347100] LustreError: 11236:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000059b59ef1 x1862458210482816/t283467841549(0) o101->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:610/0 lens 664/600 e 0 to 0 dl 1776183410 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0' [ 4927.086880] Lustre: 11236:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000792da10a x1862458210482816/t283467841549(0) o101->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:652/0 lens 664/3424 e 0 to 0 dl 1776183452 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0' [ 4933.735399] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 12:16:56 (1776183416) [ 4940.968140] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4966.555801] LDISKFS-fs (dm-0): recovery complete [ 4966.566342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4969.792522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.798513] Lustre: Skipped 8 previous similar messages [ 4973.532267] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 4975.203316] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2947 to 0x0:3009 [ 4975.207431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3939 to 0x0:3969 [ 4982.519590] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4983.879931] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5002.477312] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 12:18:05 (1776183485) [ 5011.718989] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5036.424550] LDISKFS-fs (dm-0): recovery complete [ 5036.427571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5044.594163] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5045.353881] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3971 to 0x0:4001 [ 5045.360042] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2947 to 0x0:3041 [ 5054.867385] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5056.606714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5062.198249] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5073.920191] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 12:19:16 (1776183556) [ 5121.766724] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5138.402603] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5138.412657] LustreError: Skipped 340 previous similar messages [ 5146.570505] LDISKFS-fs (dm-0): recovery complete [ 5146.577038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5154.610822] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5156.577427] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4292 to 0x0:4321 [ 5156.580070] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5252 to 0x0:5281 [ 5163.274799] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5164.872487] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5239.137628] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 12:22:02 (1776183722) [ 5246.765930] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5248.305291] Lustre: Failing over lustre-MDT0000 [ 5248.311290] Lustre: Skipped 8 previous similar messages [ 5248.475306] Lustre: server umount lustre-MDT0000 complete [ 5248.478952] Lustre: Skipped 8 previous similar messages [ 5248.483120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5248.501072] Lustre: Skipped 33 previous similar messages [ 5270.298759] LDISKFS-fs (dm-0): recovery complete [ 5270.302520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5279.656143] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5279.661882] Lustre: Skipped 8 previous similar messages [ 5282.806475] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5283.924664] Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 5283.940337] Lustre: Skipped 8 previous similar messages [ 5283.978172] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4292 to 0x0:4353 [ 5283.983047] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5283 to 0x0:5313 [ 5290.879580] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5292.164885] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5301.324181] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5302.784125] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5309.014499] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 12:23:12 (1776183792) [ 5316.684978] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5358.189944] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 5358.199352] Lustre: Skipped 3 previous similar messages [ 5360.224916] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5360.227956] Lustre: Skipped 1 previous similar message [ 5360.230492] LustreError: 17936:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000007e68eee0 x1862458211974720/t300647710728(0) o36->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:330/0 lens 66040/440 e 0 to 0 dl 1776183885 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0' [ 5402.287789] Lustre: 17936:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000324d6ac2 x1862458211974720/t300647710728(0) o36->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:373/0 lens 66040/440 e 0 to 0 dl 1776183928 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0' [ 5413.297950] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5415.171706] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 12:24:58 (1776183898) [ 5428.974450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5432.292226] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5432.300828] LustreError: Skipped 6 previous similar messages [ 5439.455329] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776183918/real 1776183918] req@00000000159a37fa x1862458231704384/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776183925 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' [ 5439.478958] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 5439.493276] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5439.506826] LustreError: Skipped 7 previous similar messages [ 5454.450995] LDISKFS-fs (dm-0): recovery complete [ 5454.454804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5456.884059] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c374e80 to 0x908d15917c377ca1 [ 5456.892492] Lustre: Skipped 7 previous similar messages [ 5456.903422] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 5456.914063] Lustre: Skipped 29 previous similar messages [ 5457.150326] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5457.153582] Lustre: Skipped 7 previous similar messages [ 5460.633805] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5463.361942] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4455 to 0x0:4481 [ 5463.361988] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5414 to 0x0:5441 [ 5468.716338] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5470.417220] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5478.481258] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 12:26:01 (1776183961) [ 5501.064627] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5535.715838] LDISKFS-fs (dm-2): recovery complete [ 5535.718393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5537.112779] Lustre: lustre-OST0000: Denying connection for new client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5537.126516] Lustre: Skipped 12 previous similar messages [ 5538.461899] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5842 to 0x0:5857 [ 5538.472896] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129 [ 5540.542616] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5576.331565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5576.478768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5576.483215] Lustre: Skipped 5 previous similar messages [ 5581.432964] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5585.278695] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5842 to 0x0:5889 [ 5585.284748] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161 [ 5589.922096] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5591.446620] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5629.727638] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 12:28:33 (1776184113) [ 5648.739383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.302354] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5663.257181] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4882 to 0x0:4897 [ 5663.258875] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5842 to 0x0:5921 [ 5692.729297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5707.257584] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5708.330901] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4882 to 0x0:4929 [ 5708.331245] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5842 to 0x0:5953 [ 5717.134913] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5718.960708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5726.619953] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 12:30:09 (1776184209) [ 5744.100908] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5744.118029] LustreError: Skipped 213 previous similar messages [ 5757.594760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5759.787393] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:193 [ 5759.791553] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5955 to 0x0:5985 [ 5761.316363] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5768.934268] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5770.315783] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5778.975902] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 12:31:02 (1776184262) [ 5788.101296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.216824] Lustre: *** cfs_fail_loc=605, val=0*** [ 5788.220100] LustreError: 172460:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=00000000216eb8c8 failed: rc = -95 [ 5788.236755] LustreError: 172460:0:(obd_config.c:774:class_setup()) setup MGS failed (-95) [ 5788.243231] LustreError: 172460:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95 [ 5788.263169] LustreError: 172460:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered [ 5788.271776] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5788.284215] LustreError: 172460:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000 [ 5788.301935] LustreError: 172460:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5795.048283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.682786] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5803.541770] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5955 to 0x0:6017 [ 5803.544254] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4931 to 0x0:4961 [ 5809.449345] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 12:31:32 (1776184292) [ 5815.573516] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5841.279223] LDISKFS-fs (dm-0): recovery complete [ 5841.281931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5848.689569] Lustre: *** cfs_fail_loc=707, val=0*** [ 5851.721307] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 5889.673455] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 5890.120593] Lustre: lustre-MDT0000: Recovery over after 0:42, of 2 clients 2 recovered and 0 were evicted. [ 5890.128923] Lustre: Skipped 7 previous similar messages [ 5890.157124] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4974 to 0x0:4993 [ 5890.158391] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6031 to 0x0:6049 [ 5895.627314] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5897.329824] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5905.759996] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 12:33:09 (1776184389) [ 5933.134922] LustreError: 6261:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms [ 5939.175110] LustreError: 6261:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 5956.081958] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 12:33:59 (1776184439) [ 5985.968885] LustreError: 43086:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms [ 5992.016049] LustreError: 43086:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake [ 6000.046901] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 12:34:43 (1776184483) [ 6028.456842] LustreError: 17936:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms [ 6033.471140] LustreError: 17936:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6035.622094] LustreError: 17936:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms [ 6045.623164] LustreError: 17936:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6064.636165] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 12:35:47 (1776184547) [ 6125.543750] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 12:36:48 (1776184608) [ 6153.255377] LustreError: 6261:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6153.679255] LustreError: 6261:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6161.406627] LustreError: 44295:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6161.410023] LustreError: 44295:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 27 previous similar messages [ 6161.839125] LustreError: 44295:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6161.852915] LustreError: 44295:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 27 previous similar messages [ 6177.683870] LustreError: 6261:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms [ 6177.696509] LustreError: 6261:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 65 previous similar messages [ 6178.119122] LustreError: 6261:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 6178.132946] LustreError: 6261:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 64 previous similar messages [ 6185.556900] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 12:37:48 (1776184668) [ 6219.479774] Lustre: DEBUG MARKER: phase 2 [ 6230.790995] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 12:38:33 (1776184713) [ 6315.777796] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 12:39:58 (1776184798) [ 6317.335980] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6319.025593] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 12:40:02 (1776184802) [ 6324.403295] Lustre: DEBUG MARKER: Started rundbench load pid=138687 ... [ 6335.548711] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6338.836806] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6341.070552] Lustre: Failing over lustre-MDT0000 [ 6341.076994] Lustre: Skipped 8 previous similar messages [ 6341.164054] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 6341.173015] Lustre: Skipped 3 previous similar messages [ 6341.391194] Lustre: server umount lustre-MDT0000 complete [ 6341.393147] Lustre: Skipped 9 previous similar messages [ 6343.135603] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6343.145410] LustreError: Skipped 6 previous similar messages [ 6343.153492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6343.172581] Lustre: Skipped 29 previous similar messages [ 6344.677811] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6344.708611] LustreError: Skipped 76 previous similar messages [ 6351.263618] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776184830/real 1776184830] req@0000000078b4c1dc x1862458232035392/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776184837 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 6351.287891] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6351.294132] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6351.301537] LustreError: Skipped 4 previous similar messages [ 6364.456126] LDISKFS-fs (dm-0): recovery complete [ 6364.459016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.737145] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c389f36 to 0x908d15917c390327 [ 6368.752140] Lustre: Skipped 4 previous similar messages [ 6368.769107] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 6368.785491] Lustre: Skipped 30 previous similar messages [ 6369.059489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6369.069866] Lustre: Skipped 5 previous similar messages [ 6369.125334] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6369.135893] Lustre: Skipped 7 previous similar messages [ 6370.369366] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6370.374657] Lustre: Skipped 8 previous similar messages [ 6373.878915] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6375.956503] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5075 to 0x0:5121 [ 6375.962187] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6150 to 0x0:6177 [ 6385.519477] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6387.267464] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6398.286108] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6401.354201] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6428.755712] LDISKFS-fs (dm-1): recovery complete [ 6428.760646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6434.346323] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:33 [ 6434.346461] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:198 to 0x280000400:225 [ 6435.347727] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6445.453652] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6447.182356] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6477.929413] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 12:42:40 (1776184960) [ 6607.516848] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6619.898536] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 6622.090728] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 6622.104161] Lustre: Skipped 1 previous similar message [ 6649.159815] LDISKFS-fs (dm-0): recovery complete [ 6649.171511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6669.380432] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6673.612344] Lustre: lustre-MDT0000: Recovery over after 0:09, of 2 clients 2 recovered and 0 were evicted. [ 6673.625714] Lustre: Skipped 2 previous similar messages [ 6673.701840] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5478 to 0x0:5505 [ 6673.702546] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6533 to 0x0:6561 [ 6680.198090] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6681.863591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6779.769964] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 12:47:42 (1776185262) [ 6909.086578] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6921.845446] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 6924.126236] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.30@tcp (stopping) [ 6945.766487] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6945.780164] LustreError: Skipped 122 previous similar messages [ 6956.412733] LDISKFS-fs (dm-1): recovery complete [ 6956.419064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.499272] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 6965.568279] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:614 to 0x280000400:641 [ 6965.583300] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:422 to 0x2c0000400:449 [ 6976.790200] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6978.871559] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6989.828765] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 12:51:12 (1776185472) [ 7120.079267] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7132.424459] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 7134.888388] Lustre: Failing over lustre-MDT0000 [ 7134.891473] Lustre: Skipped 3 previous similar messages [ 7135.143467] Lustre: server umount lustre-MDT0000 complete [ 7135.148355] Lustre: Skipped 3 previous similar messages [ 7136.224584] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7136.242343] LustreError: Skipped 3 previous similar messages [ 7136.247501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.257686] Lustre: Skipped 13 previous similar messages [ 7143.903162] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776185622/real 1776185622] req@00000000fc34fa5d x1862458233957760/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776185629 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 7143.948135] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7143.961219] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7143.975510] LustreError: Skipped 1 previous similar message [ 7161.132251] LDISKFS-fs (dm-0): recovery complete [ 7161.141967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7176.673772] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c3d3e59 to 0x908d15917c45f951 [ 7176.685421] Lustre: Skipped 1 previous similar message [ 7176.711933] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 7176.721234] Lustre: Skipped 15 previous similar messages [ 7176.880671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7176.885533] Lustre: Skipped 3 previous similar messages [ 7176.915200] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7176.927741] Lustre: Skipped 3 previous similar messages [ 7178.102610] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7178.112703] Lustre: Skipped 3 previous similar messages [ 7181.911540] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7188.442131] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7169 to 0x0:7201 [ 7188.444150] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6113 to 0x0:6145 [ 7195.122593] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7197.117584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7207.041716] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 12:54:49 (1776185689) [ 7219.125348] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7220.837922] Lustre: lustre-OST0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnecting [ 7220.844946] Lustre: Skipped 1 previous similar message [ 7222.143533] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7253.940024] LDISKFS-fs (dm-2): recovery complete [ 7253.958689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7255.868204] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7244 to 0x0:7265 [ 7255.871786] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:648 to 0x280000400:673 [ 7260.869286] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7274.731606] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7276.949875] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7287.451515] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 12:56:10 (1776185770) [ 7419.445529] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7428.831979] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7441.444608] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 7444.486952] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7444.495628] Lustre: Skipped 5 previous similar messages [ 7478.536671] LDISKFS-fs (dm-1): recovery complete [ 7478.541606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7545.831329] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7545.849145] LustreError: Skipped 155 previous similar messages [ 7585.693574] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7609.872310] LDISKFS-fs (dm-0): recovery complete [ 7609.875931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7626.023931] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7627.382538] Lustre: lustre-MDT0001: Recovery over after 0:45, of 2 clients 2 recovered and 0 were evicted. [ 7627.398401] Lustre: Skipped 3 previous similar messages [ 7627.521130] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:648 to 0x280000400:705 [ 7627.522732] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:456 to 0x2c0000400:481 [ 7635.180543] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6189 to 0x0:6209 [ 7635.181100] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7244 to 0x0:7297 [ 7641.533075] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7643.622459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7645.352350] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7654.585917] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 13:02:17 (1776186137) [ 7666.882099] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7694.524822] LDISKFS-fs (dm-0): recovery complete [ 7694.530652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.380942] LustreError: 198836:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.204.130@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (000000004af4398d) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7698.557675] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7701.410675] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7705.710927] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 7705.786712] Lustre: 195655:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000001f37af8 x1862458217953728/t335007449632(335007449632) o101->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:377/0 lens 648/3424 e 0 to 0 dl 1776186197 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7705.955204] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6189 to 0x0:6241 [ 7705.958906] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7299 to 0x0:7329 [ 7713.870263] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7717.021442] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7728.607728] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 13:03:31 (1776186211) [ 7736.658041] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7739.157238] Lustre: Failing over lustre-MDT0000 [ 7739.158980] Lustre: Skipped 4 previous similar messages [ 7739.366434] Lustre: server umount lustre-MDT0000 complete [ 7739.373383] Lustre: Skipped 4 previous similar messages [ 7741.921564] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7741.937898] LustreError: Skipped 5 previous similar messages [ 7741.947763] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7741.973646] Lustre: Skipped 14 previous similar messages [ 7751.647500] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186229/real 1776186229] req@000000003f83362e x1862458235018624/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776186236 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0' [ 7751.658851] Lustre: 3359:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7751.662539] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7751.676521] LustreError: Skipped 2 previous similar messages [ 7763.272411] LDISKFS-fs (dm-0): recovery complete [ 7763.276724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7770.682867] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7770.692532] LustreError: 195654:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000076abf07b x1862458217953728/t335007449632(335007449632) o101->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:442/0 lens 648/600 e 0 to 0 dl 1776186262 ref 1 fl Interpret:/4/0 rc 301/0 job:'lfs.0' [ 7774.073163] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7777.918655] Lustre: lustre-MDT0000: Client 543b08dc-b2ae-46cd-aa25-e10cda298cc8 (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [ 7777.932348] Lustre: 195656:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@000000004b70249f x1862458217953728/t335007449632(335007449632) o101->543b08dc-b2ae-46cd-aa25-e10cda298cc8@192.168.204.30@tcp:449/0 lens 648/3424 e 0 to 0 dl 1776186269 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0' [ 7778.044515] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to (at 0@lo) [ 7778.060650] Lustre: Skipped 22 previous similar messages [ 7778.104955] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6273 [ 7778.104955] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7299 to 0x0:7361 [ 7786.273552] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7788.359039] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7800.314752] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 13:04:42 (1776186282) [ 7830.464995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7840.231956] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c49f934 to 0x908d15917c49fdb7 [ 7840.242711] Lustre: Skipped 3 previous similar messages [ 7840.511350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7840.516525] Lustre: Skipped 5 previous similar messages [ 7840.551975] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7840.558599] Lustre: Skipped 5 previous similar messages [ 7845.058375] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7845.860470] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 7845.871502] Lustre: Skipped 5 previous similar messages [ 7845.920166] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6305 [ 7854.263433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7855.961945] Lustre: lustre-OST0000: Denying connection for new client 7927cff7-3970-4e93-b5bd-cc9c2b96081b (at 192.168.204.30@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7856.281078] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:648 to 0x280000400:737 [ 7856.305521] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7299 to 0x0:7393 [ 7858.758743] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7868.620816] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:05:51 (1776186351) [ 7869.901073] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7869.913723] LustreError: 8247:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000055cec5c0 x1862458235055296/t352187318279(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:541/0 lens 1056/4320 e 0 to 0 dl 1776186361 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7876.983194] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7877.091178] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7905.927222] LDISKFS-fs (dm-0): recovery complete [ 7905.933559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.700189] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7927.414989] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6337 [ 7927.415309] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7425 [ 7936.481373] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7938.242177] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7948.736064] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:07:11 (1776186431) [ 7950.118047] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7950.130558] LustreError: 185264:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@000000000ecb7011 x1862458235081280/t356482285578(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:621/0 lens 2200/4320 e 0 to 0 dl 1776186441 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 7956.455800] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7958.086616] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7966.149158] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7992.240089] LDISKFS-fs (dm-1): recovery complete [ 7992.246278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7997.497977] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 7997.984455] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:492 to 0x2c0000400:513 [ 7997.986673] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:748 to 0x280000400:769 [ 8009.802461] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8012.395373] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8024.313294] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:08:27 (1776186507) [ 8025.939545] LustreError: 8246:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000b2f44e1c x1862458235103872/t356482285597(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:697/0 lens 2200/4320 e 0 to 0 dl 1776186517 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 8032.252142] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8037.096450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8047.320185] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8073.677607] LDISKFS-fs (dm-0): recovery complete [ 8073.681350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8081.467215] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8082.651092] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7457 [ 8082.653930] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6369 [ 8091.646255] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8093.306749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8122.457264] LDISKFS-fs (dm-1): recovery complete [ 8122.460748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.278791] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8128.036961] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:524 to 0x2c0000400:545 [ 8128.037337] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:780 to 0x280000400:801 [ 8137.186742] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8139.048316] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8148.525189] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:10:31 (1776186631) [ 8149.778603] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8149.780925] Lustre: Skipped 1 previous similar message [ 8157.157221] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8161.249684] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8169.438859] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8171.636494] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 8171.650260] Lustre: Skipped 8 previous similar messages [ 8172.523052] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8172.539340] LustreError: Skipped 334 previous similar messages [ 8175.495696] LustreError: 9953:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776186661 with bad export cookie 10416005227777497961 [ 8175.512949] LustreError: 9953:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8204.558510] LDISKFS-fs (dm-0): recovery complete [ 8204.562528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.917634] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8238.235581] LDISKFS-fs (dm-1): recovery complete [ 8238.237943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8243.219062] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8244.706981] Lustre: lustre-MDT0000: Recovery over after 0:26, of 2 clients 2 recovered and 0 were evicted. [ 8244.720136] Lustre: Skipped 9 previous similar messages [ 8244.764989] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7489 [ 8244.764989] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6401 [ 8244.780620] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:556 to 0x2c0000400:577 [ 8244.784135] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:812 to 0x280000400:833 [ 8253.937739] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8255.529962] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8257.079625] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8266.559767] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:12:29 (1776186749) [ 8268.036711] LustreError: 216062:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000aea8d237 x1862458218779776/t30064771142(0) o36->7927cff7-3970-4e93-b5bd-cc9c2b96081b@192.168.204.30@tcp:184/0 lens 560/448 e 0 to 0 dl 1776186759 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0' [ 8268.069654] LustreError: 216062:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 1 previous similar message [ 8275.043172] Lustre: lustre-MDT0001: Client 7927cff7-3970-4e93-b5bd-cc9c2b96081b (at 192.168.204.30@tcp) reconnecting [ 8275.057557] Lustre: 214219:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000bcf39509 x1862458218779776/t30064771142(0) o36->7927cff7-3970-4e93-b5bd-cc9c2b96081b@192.168.204.30@tcp:191/0 lens 560/448 e 0 to 0 dl 1776186766 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8278.322044] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8303.657424] LDISKFS-fs (dm-0): recovery complete [ 8303.666690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8314.567653] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8315.027613] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7521 [ 8315.040517] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6433 [ 8325.185908] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8326.804217] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8336.088154] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:13:38 (1776186818) [ 8344.505404] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8346.783030] Lustre: Failing over lustre-MDT0001 [ 8346.785994] Lustre: Skipped 9 previous similar messages [ 8346.936201] Lustre: server umount lustre-MDT0001 complete [ 8346.938547] Lustre: Skipped 9 previous similar messages [ 8350.699254] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8350.705787] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8350.715743] LustreError: Skipped 9 previous similar messages [ 8350.753352] Lustre: Skipped 38 previous similar messages [ 8371.927709] LDISKFS-fs (dm-1): recovery complete [ 8371.938271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.522989] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8377.898802] Lustre: 216196:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000e8ead81f x1862458218797568/t30064771212(0) o36->7927cff7-3970-4e93-b5bd-cc9c2b96081b@192.168.204.30@tcp:331/0 lens 560/448 e 0 to 0 dl 1776186906 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8377.924654] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:598 to 0x2c0000400:641 [ 8377.926799] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:854 to 0x280000400:897 [ 8386.442603] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8388.845632] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8401.961810] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:14:44 (1776186884) [ 8410.257081] Lustre: lustre-MDT0001: Client 7927cff7-3970-4e93-b5bd-cc9c2b96081b (at 192.168.204.30@tcp) reconnecting [ 8410.268493] Lustre: 216062:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@00000000924def0e x1862458218814208/t34359738438(0) o36->7927cff7-3970-4e93-b5bd-cc9c2b96081b@192.168.204.30@tcp:327/0 lens 560/448 e 0 to 0 dl 1776186902 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8413.580040] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8420.955824] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8430.560232] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776186909/real 1776186909] req@00000000e1616cff x1862458235223936/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776186916 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' [ 8430.596732] Lustre: 3357:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 8430.610666] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8430.617269] LustreError: Skipped 5 previous similar messages [ 8445.364208] LDISKFS-fs (dm-0): recovery complete [ 8445.371000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8447.984204] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c4a3922 to 0x908d15917c4a50c2 [ 8448.009755] Lustre: Skipped 4 previous similar messages [ 8448.023325] Lustre: MGC192.168.204.130@tcp: Connection restored to (at 0@lo) [ 8448.033193] Lustre: Skipped 41 previous similar messages [ 8448.337117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8448.347726] Lustre: Skipped 9 previous similar messages [ 8448.432472] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8448.450693] Lustre: Skipped 9 previous similar messages [ 8449.126542] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8449.146139] Lustre: Skipped 9 previous similar messages [ 8453.826503] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7553 [ 8453.828318] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6465 [ 8454.179861] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8464.139746] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8465.722880] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8494.798084] LDISKFS-fs (dm-1): recovery complete [ 8494.804682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8500.178988] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8500.261895] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:908 to 0x280000400:929 [ 8500.278138] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:652 to 0x2c0000400:673 [ 8510.459225] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8512.088889] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8522.097756] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:16:45 (1776187005) [ 8523.051190] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8523.057813] Lustre: Skipped 3 previous similar messages [ 8529.504348] Lustre: lustre-MDT0001: Client 7927cff7-3970-4e93-b5bd-cc9c2b96081b (at 192.168.204.30@tcp) reconnecting [ 8529.530524] Lustre: 214220:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000090fdd081 x1862458218836480/t38654705734(0) o36->7927cff7-3970-4e93-b5bd-cc9c2b96081b@192.168.204.30@tcp:446/0 lens 560/448 e 0 to 0 dl 1776187021 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0' [ 8533.992199] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8542.071696] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8547.693475] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187033 with bad export cookie 10416005227777511618 [ 8547.711521] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8572.920968] LDISKFS-fs (dm-0): recovery complete [ 8572.923757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8585.726308] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8612.136924] LDISKFS-fs (dm-1): recovery complete [ 8612.142345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8617.109928] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8618.392477] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7585 [ 8618.393239] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6497 [ 8618.470360] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:940 to 0x280000400:961 [ 8618.471169] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:684 to 0x2c0000400:705 [ 8626.366152] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8628.313470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8629.984583] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8640.408737] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:18:43 (1776187123) [ 8641.622803] LustreError: 185259:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@00000000a5983d51 x1862458235285504/t377957122063(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:558/0 lens 640/4320 e 0 to 0 dl 1776187133 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0' [ 8641.650503] LustreError: 185259:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 3 previous similar messages [ 8648.674136] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8648.845463] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8673.899809] LDISKFS-fs (dm-1): recovery complete [ 8673.910897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.558586] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8679.532212] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:993 [ 8679.535211] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:737 [ 8687.916542] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8689.880306] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8699.714813] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:19:42 (1776187182) [ 8708.579936] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8709.883503] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8735.847464] LDISKFS-fs (dm-0): recovery complete [ 8735.853875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8742.564928] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8743.115996] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6529 [ 8743.119498] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7617 [ 8755.392445] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8757.478433] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8768.196667] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:20:50 (1776187250) [ 8776.673482] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8778.594210] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8786.305865] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8790.641721] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8790.658562] LustreError: Skipped 228 previous similar messages [ 8811.666601] LDISKFS-fs (dm-0): recovery complete [ 8811.668792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.491675] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8821.958411] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7649 [ 8821.965587] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6561 [ 8833.663568] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8835.612710] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8863.471374] LDISKFS-fs (dm-1): recovery complete [ 8863.479934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8868.623299] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8869.372108] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8869.392267] Lustre: Skipped 10 previous similar messages [ 8869.476826] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1025 [ 8869.482122] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:769 [ 8880.544794] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8882.791321] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8892.616668] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:22:55 (1776187375) [ 8901.606170] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8903.039925] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8910.292964] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8916.051745] LustreError: 9953:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187401 with bad export cookie 10416005227777521964 [ 8916.062446] LustreError: 9953:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8943.934886] LDISKFS-fs (dm-0): recovery complete [ 8943.939727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8953.215944] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8975.935170] LDISKFS-fs (dm-1): recovery complete [ 8975.948509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8976.319709] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 8976.329964] LustreError: Skipped 8 previous similar messages [ 8980.847712] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 8981.769783] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:801 [ 8981.772936] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1057 [ 8981.935432] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6593 [ 8981.935544] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7681 [ 8990.482740] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8991.775492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8993.215634] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9002.006386] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:24:44 (1776187484) [ 9009.777431] Lustre: lustre-MDT0001: Client 7927cff7-3970-4e93-b5bd-cc9c2b96081b (at 192.168.204.30@tcp) reconnecting [ 9009.819754] Lustre: 237862:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno req@0000000055eca7e1 x1862458218894528/t55834574850(0) o36->7927cff7-3970-4e93-b5bd-cc9c2b96081b@192.168.204.30@tcp:171/0 lens 496/2888 e 0 to 0 dl 1776187501 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0' [ 9013.787568] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9016.058921] Lustre: Failing over lustre-MDT0000 [ 9016.063589] Lustre: Skipped 10 previous similar messages [ 9016.264100] Lustre: server umount lustre-MDT0000 complete [ 9016.272181] Lustre: Skipped 10 previous similar messages [ 9017.316435] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.331601] Lustre: Skipped 37 previous similar messages [ 9037.569079] LDISKFS-fs (dm-0): recovery complete [ 9037.581962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9047.173790] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7713 [ 9047.174911] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6625 [ 9047.472323] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9060.065832] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9061.932828] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9071.428727] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:25:54 (1776187554) [ 9073.049067] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 9073.054825] Lustre: Skipped 5 previous similar messages [ 9080.222861] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9104.266445] LDISKFS-fs (dm-1): recovery complete [ 9104.269540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9104.615950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9104.621520] Lustre: Skipped 10 previous similar messages [ 9104.647645] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 9104.659459] Lustre: Skipped 10 previous similar messages [ 9106.018513] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9106.034245] Lustre: Skipped 10 previous similar messages [ 9109.296294] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9109.991850] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to (at 0@lo) [ 9110.005491] Lustre: Skipped 46 previous similar messages [ 9110.082340] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:833 [ 9110.085796] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1089 [ 9120.643190] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9122.568895] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9132.232326] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:26:54 (1776187614) [ 9141.356852] Lustre: lustre-MDT0001: Client 7927cff7-3970-4e93-b5bd-cc9c2b96081b (at 192.168.204.30@tcp) reconnecting [ 9141.375698] Lustre: Skipped 1 previous similar message [ 9141.862405] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9148.826374] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9162.655672] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776187641/real 1776187641] req@00000000b549ae74 x1862458235434368/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776187648 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0' [ 9162.679625] Lustre: 3358:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 22 previous similar messages [ 9162.683783] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9162.696504] LustreError: Skipped 5 previous similar messages [ 9174.030146] LDISKFS-fs (dm-0): recovery complete [ 9174.040512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9180.135239] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c4a8b54 to 0x908d15917c4a9675 [ 9180.149521] Lustre: Skipped 5 previous similar messages [ 9184.925811] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9186.007489] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6657 [ 9186.008544] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7745 [ 9194.842342] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9196.852748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9223.954674] LDISKFS-fs (dm-1): recovery complete [ 9223.962214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9228.666847] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9229.371173] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:865 [ 9229.372388] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1121 [ 9237.551694] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9239.146411] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9247.214967] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 13:28:50 (1776187730) [ 9248.449663] LustreError: 239209:0:(ldlm_lib.c:3244:target_send_reply_msg()) @@@ dropping reply req@0000000092f74295 x1862458218922880/t64424509442(0) o36->7927cff7-3970-4e93-b5bd-cc9c2b96081b@192.168.204.30@tcp:410/0 lens 496/456 e 0 to 0 dl 1776187740 ref 1 fl Interpret:/0/0 rc 0/0 job:'rmdir.0' [ 9248.479269] LustreError: 239209:0:(ldlm_lib.c:3244:target_send_reply_msg()) Skipped 6 previous similar messages [ 9256.680615] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9265.127538] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9271.611590] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776187757 with bad export cookie 10416005227777529461 [ 9271.628506] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9294.452119] LDISKFS-fs (dm-0): recovery complete [ 9294.455581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9309.697874] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9333.885599] LDISKFS-fs (dm-1): recovery complete [ 9333.888699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9338.294318] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9340.211469] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6243 to 0x0:6689 [ 9340.211649] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7395 to 0x0:7777 [ 9340.267647] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1153 [ 9340.269693] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:897 [ 9348.865345] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9350.602438] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9352.114498] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9362.135975] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 13:30:45 (1776187845) [ 9364.101988] Lustre: 251740:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 7927cff7-3970-4e93-b5bd-cc9c2b96081b at adminstrative request [ 9375.979530] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 13:30:58 (1776187858) [ 9391.710417] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9391.725767] LustreError: Skipped 231 previous similar messages [ 9402.936551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9415.320506] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9415.892276] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6741 to 0x0:6785 [ 9415.896849] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7828 to 0x0:7873 [ 9427.934307] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9430.136356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9441.943334] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 13:32:05 (1776187925) [ 9483.576401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9485.655841] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9485.673903] Lustre: Skipped 9 previous similar messages [ 9485.681405] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1185 [ 9485.699843] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7974 to 0x0:8001 [ 9488.496540] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9498.198817] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9500.316438] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9510.451073] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 13:33:13 (1776187993) [ 9524.882376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9530.260152] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9530.456568] Lustre: lustre-OST0000: deleting orphan objects from 0x0:7974 to 0x0:8033 [ 9530.458842] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6741 to 0x0:6817 [ 9540.488845] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 13:33:43 (1776188023) [ 9548.209654] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9573.023130] LDISKFS-fs (dm-2): recovery complete [ 9573.029944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9575.043948] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8035 to 0x0:8065 [ 9575.052021] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1217 [ 9578.178599] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9588.593451] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9590.283546] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9599.211885] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 13:34:42 (1776188082) [ 9607.468501] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9611.232158] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9611.239462] LustreError: Skipped 7 previous similar messages [ 9633.512594] LDISKFS-fs (dm-2): recovery complete [ 9633.515614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9635.593992] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.30@tcp inode [0x20002f201:0x5:0x0] object 0x0:8066 extent [0-4194303]: client csum 8f788298, server csum 22323649 [ 9635.713260] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8067 to 0x0:8097 [ 9635.723389] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1249 [ 9637.600983] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9645.831949] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9647.471569] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9656.123792] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 13:35:38 (1776188138) [ 9663.679846] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9672.726806] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9681.515826] Lustre: Failing over lustre-MDT0000 [ 9681.524527] Lustre: Skipped 10 previous similar messages [ 9681.542957] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 9681.567553] Lustre: Skipped 5 previous similar messages [ 9681.872795] Lustre: server umount lustre-MDT0000 complete [ 9681.875629] Lustre: Skipped 10 previous similar messages [ 9683.938654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9683.966558] Lustre: Skipped 30 previous similar messages [ 9685.959910] LustreError: 6246:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188171 with bad export cookie 10416005227777556131 [ 9706.989620] LDISKFS-fs (dm-0): recovery complete [ 9706.992662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.784874] Lustre: MGC192.168.204.130@tcp: Connection restored to 192.168.204.130@tcp (at 0@lo) [ 9718.801307] Lustre: Skipped 33 previous similar messages [ 9719.121295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9719.128975] Lustre: Skipped 9 previous similar messages [ 9719.196768] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9719.210650] Lustre: Skipped 9 previous similar messages [ 9720.663346] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9720.670619] Lustre: Skipped 9 previous similar messages [ 9725.579236] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9725.600892] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6741 to 0x0:6849 [ 9749.948508] LDISKFS-fs (dm-2): recovery complete [ 9749.953037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9752.210852] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1281 [ 9755.147296] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9772.760408] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 13:37:35 (1776188255) [ 9798.047370] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1776188276/real 1776188276] req@00000000e5bb73c4 x1862458235612544/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1776188283 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0' [ 9798.086856] Lustre: 3356:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 9798.096145] LustreError: 166-1: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9798.112154] LustreError: Skipped 4 previous similar messages [ 9807.591206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9814.515780] Lustre: Evicted from MGS (at 192.168.204.130@tcp) after server handle changed from 0x908d15917c4b10b6 to 0x908d15917c4b2585 [ 9814.524338] Lustre: Skipped 4 previous similar messages [ 9819.800984] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9820.249965] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6741 to 0x0:6881 [ 9828.358512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9833.028561] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9835.592969] Lustre: lustre-OST0000: Denying connection for new client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 9840.751514] Lustre: lustre-OST0000: Denying connection for new client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 9845.862301] Lustre: lustre-OST0000: Denying connection for new client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 9850.977926] Lustre: lustre-OST0000: Denying connection for new client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 9856.098669] Lustre: lustre-OST0000: Denying connection for new client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:42 [ 9866.341287] Lustre: lustre-OST0000: Denying connection for new client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 9866.360929] Lustre: Skipped 1 previous similar message [ 9886.830443] Lustre: lustre-OST0000: Denying connection for new client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:12 [ 9886.847428] Lustre: Skipped 3 previous similar messages [ 9899.000268] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9899.016744] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9899.044930] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1313 [ 9899.054532] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8148 to 0x0:8169 [ 9905.009404] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 59 sec [ 9914.632336] Lustre: DEBUG MARKER: free_before: 7646580 free_after: 7646580 [ 9920.913239] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 13:40:03 (1776188403) [ 9944.516058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9946.640762] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:716 to 0x2c0000400:929 [ 9946.641777] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6884 to 0x0:6913 [ 9948.680847] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9958.080740] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 13:40:41 (1776188441) [ 9978.924467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9980.738386] LustreError: 268410:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9980.744977] LustreError: 268410:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [ 9981.279140] Lustre: *** cfs_fail_loc=715, val=0*** [ 9982.599799] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [ 9988.064501] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:57 [ 9988.084645] Lustre: Skipped 1 previous similar message [ 9989.087292] Lustre: *** cfs_fail_loc=715, val=0*** [ 9989.099055] Lustre: Skipped 2 previous similar messages [ 9995.233040] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:50 [ 9995.248810] Lustre: Skipped 2 previous similar messages [ 9996.256778] Lustre: *** cfs_fail_loc=715, val=0*** [ 9996.263205] Lustre: Skipped 2 previous similar messages [10001.506234] Lustre: lustre-OST0000: Client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp) reconnected, waiting for 3 clients in recovery for 0:44 [10001.525618] Lustre: Skipped 1 previous similar message [10002.591907] Lustre: *** cfs_fail_loc=715, val=0*** [10002.596711] Lustre: Skipped 2 previous similar messages [10008.670106] Lustre: lustre-OST0000: Client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp) reconnected, waiting for 3 clients in recovery for 0:37 [10008.692174] Lustre: Skipped 2 previous similar messages [10009.759224] Lustre: *** cfs_fail_loc=715, val=0*** [10009.765860] Lustre: Skipped 2 previous similar messages [10015.842781] Lustre: lustre-OST0000: Client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp) reconnected, waiting for 3 clients in recovery for 0:30 [10015.878766] Lustre: Skipped 2 previous similar messages [10017.955615] Lustre: *** cfs_fail_loc=715, val=0*** [10017.965796] Lustre: Skipped 3 previous similar messages [10020.839591] LustreError: 268410:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [10020.851144] LustreError: 268410:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages [10020.876338] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:972 to 0x280000400:1345 [10020.884307] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8173 to 0x0:8201 [10028.538574] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10030.302371] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10038.773160] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 13:42:01 (1776188521) [10045.410589] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10045.420094] LustreError: Skipped 229 previous similar messages [10062.801876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10071.141915] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (not set up) [10073.824141] Lustre: *** cfs_fail_loc=715, val=0*** [10073.827160] Lustre: Skipped 1 previous similar message [10075.171560] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 [10076.653342] LustreError: 269975:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 sleeping for 80000ms [10079.845468] Lustre: lustre-MDT0000: Client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:58 [10079.869216] Lustre: Skipped 2 previous similar messages [10083.807600] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10098.147145] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10098.152222] Lustre: Skipped 1 previous similar message [10101.345783] Lustre: lustre-MDT0000: Client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:36 [10101.352814] Lustre: Skipped 2 previous similar messages [10108.511891] Lustre: *** cfs_fail_loc=715, val=0*** [10108.519791] Lustre: Skipped 10 previous similar messages [10118.633046] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10118.644979] Lustre: Skipped 2 previous similar messages [10136.159221] Lustre: lustre-MDT0000: Client b39dad18-4c7b-4948-b3bd-e6bf257ae819 (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:01 [10136.170146] Lustre: Skipped 4 previous similar messages [10143.330238] Lustre: lustre-MDT0000: Recovery already passed deadline 0:05. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [10149.479639] Lustre: lustre-MDT0000: Recovery already passed deadline 0:11. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [10153.441122] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10153.458685] Lustre: Skipped 4 previous similar messages [10156.649049] Lustre: lustre-MDT0000: Recovery already passed deadline 0:18. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [10156.727107] LustreError: 269975:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 715 awake [10156.744703] Lustre: 269975:0:(ldlm_lib.c:2849:target_recovery_thread()) too long recovery - read logs [10156.755109] LustreError: dumping log to /tmp/lustre-log.1776188642.269975 [10156.826975] Lustre: lustre-MDT0000: Recovery over after 1:24, of 2 clients 2 recovered and 0 were evicted. [10156.840652] Lustre: Skipped 9 previous similar messages [10156.897922] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8212 to 0x0:8233 [10156.898519] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6924 to 0x0:6945 [10162.714854] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10164.364992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10172.864110] Lustre: DEBUG MARKER: == replay-single test complete, duration 9875 sec ======== 13:44:15 (1776188655) [10199.727105] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776188685 with bad export cookie 10416005227777570495 [10199.744162] LustreError: 6247:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 6 previous similar messages [10242.200414] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [10245.840542] Key type lgssc unregistered [10246.135524] LNet: 273162:0:(lib-ptl.c:992:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10246.146308] LNet: Removed LNI 192.168.204.130@tcp [10247.210436] Key type .llcrypt unregistered [10247.218680] Key type ._llcrypt unregistered