[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491653519 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.003013] kvm-guest: setup PV IPIs [ 0.006319] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007028] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009136] LSM: Security Framework initializing [ 0.010033] Yama: becoming mindful. [ 0.011023] SELinux: Initializing. [ 0.011742] *** VALIDATE selinux *** [ 0.019618] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024224] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025142] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029045] *** VALIDATE tmpfs *** [ 0.030303] *** VALIDATE proc *** [ 0.031214] *** VALIDATE cgroup *** [ 0.032006] *** VALIDATE cgroup2 *** [ 0.033232] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036023] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040305] debug: unmapping init [mem 0xffffffffa1059000-0xffffffffa1060fff] [ 0.043218] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044698] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045017] ... version: 2 [ 0.045839] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.046940] ... value mask: 0000ffffffffffff [ 0.047011] ... max period: 00007fffffffffff [ 0.048011] ... fixed-purpose events: 3 [ 0.048933] ... event mask: 000000070000000f [ 0.049244] rcu: Hierarchical SRCU implementation. [ 0.051494] smp: Bringing up secondary CPUs ... [ 0.052479] x86: Booting SMP configuration: [ 0.053024] .... node #0, CPUs: #1 #2 #3 [ 0.063221] smp: Brought up 1 node, 4 CPUs [ 0.064945] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.104945] node 0 deferred pages initialised in 38ms [ 0.109592] devtmpfs: initialized [ 0.110362] x86/mm: Memory block size: 128MB [ 0.117279] gcov: version magic: 0x41383552 [ 0.122936] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.123123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.124421] pinctrl core: initialized pinctrl subsystem [ 0.125307] [ 0.125802] ************************************************************* [ 0.126137] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127015] ** ** [ 0.128016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.129015] ** ** [ 0.130015] ** This means that this kernel is built to expose internal ** [ 0.131016] ** IOMMU data structures, which may compromise security on ** [ 0.132018] ** your system. ** [ 0.133014] ** ** [ 0.134105] ** If you see this message and you are not debugging the ** [ 0.135010] ** kernel, report this immediately to your vendor! ** [ 0.136012] ** ** [ 0.137013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138011] ************************************************************* [ 0.139788] NET: Registered protocol family 16 [ 0.140532] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.141094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.142060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.143656] cpuidle: using governor menu [ 0.144925] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.146616] PCI: Using configuration type 1 for base access [ 0.148217] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.153256] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.154014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.157107] cryptd: max_cpu_qlen set to 1000 [ 0.159244] ACPI: Added _OSI(Module Device) [ 0.160013] ACPI: Added _OSI(Processor Device) [ 0.161012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.163009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.168273] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.174737] ACPI: Interpreter enabled [ 0.175049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.176056] ACPI: Using IOAPIC for interrupt routing [ 0.177169] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.178401] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.189354] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.191030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.193015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.196067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.200604] acpiphp: Slot [2] registered [ 0.202081] acpiphp: Slot [5] registered [ 0.203083] acpiphp: Slot [6] registered [ 0.204070] acpiphp: Slot [7] registered [ 0.205152] acpiphp: Slot [8] registered [ 0.206300] acpiphp: Slot [9] registered [ 0.208079] acpiphp: Slot [10] registered [ 0.209082] acpiphp: Slot [3] registered [ 0.210066] acpiphp: Slot [4] registered [ 0.211123] acpiphp: Slot [11] registered [ 0.213070] acpiphp: Slot [12] registered [ 0.214099] acpiphp: Slot [13] registered [ 0.215070] acpiphp: Slot [14] registered [ 0.216091] acpiphp: Slot [15] registered [ 0.217064] acpiphp: Slot [16] registered [ 0.219062] acpiphp: Slot [17] registered [ 0.220065] acpiphp: Slot [18] registered [ 0.221156] acpiphp: Slot [19] registered [ 0.222096] acpiphp: Slot [20] registered [ 0.223074] acpiphp: Slot [21] registered [ 0.224231] acpiphp: Slot [22] registered [ 0.226082] acpiphp: Slot [23] registered [ 0.227065] acpiphp: Slot [24] registered [ 0.228067] acpiphp: Slot [25] registered [ 0.229078] acpiphp: Slot [26] registered [ 0.230136] acpiphp: Slot [27] registered [ 0.232093] acpiphp: Slot [28] registered [ 0.233056] acpiphp: Slot [29] registered [ 0.234075] acpiphp: Slot [30] registered [ 0.235084] acpiphp: Slot [31] registered [ 0.236045] PCI host bridge to bus 0000:00 [ 0.237012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.239041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.241017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.243016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.245067] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.247019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.249244] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.251675] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.255279] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.264752] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.270049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.272024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.274021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.277019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.279616] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.281781] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.283040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.285768] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.290015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.304019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.307969] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.314791] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.323025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.324000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.345018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.356170] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.364018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.371018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.391020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.407771] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.416017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.422017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.436019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.447368] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.454020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.461036] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.479023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.492425] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.500017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.507019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.526017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.539977] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.548021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.560023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.578017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.590903] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.593363] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.596599] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.599798] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.602630] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.607326] iommu: Default domain type: Passthrough [ 0.609508] SCSI subsystem initialized [ 0.611156] ACPI: bus type USB registered [ 0.613091] usbcore: registered new interface driver usbfs [ 0.615095] usbcore: registered new interface driver hub [ 0.617083] usbcore: registered new device driver usb [ 0.619163] pps_core: LinuxPPS API ver. 1 registered [ 0.621013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.624059] PTP clock support registered [ 0.626184] EDAC MC: Ver: 3.0.0 [ 0.628015] PCI: Using ACPI for IRQ routing [ 0.628960] NetLabel: Initializing [ 0.630010] NetLabel: domain hash size = 128 [ 0.631000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.633304] NetLabel: unlabeled traffic allowed by default [ 0.634631] vgaarb: loaded [ 0.636746] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.638024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.643451] clocksource: Switched to clocksource kvm-clock [ 0.791208] VFS: Disk quotas dquot_6.6.0 [ 0.792129] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.793623] *** VALIDATE ramfs *** [ 0.794414] *** VALIDATE hugetlbfs *** [ 0.797207] pnp: PnP ACPI init [ 0.799169] pnp: PnP ACPI: found 6 devices [ 0.815310] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.817758] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.819205] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.820686] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.822423] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.824220] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.826329] NET: Registered protocol family 2 [ 0.828431] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.833163] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.835961] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.840950] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.844162] TCP: Hash tables configured (established 65536 bind 65536) [ 0.846621] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.848985] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.850982] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853332] NET: Registered protocol family 1 [ 0.855324] RPC: Registered named UNIX socket transport module. [ 0.856746] RPC: Registered udp transport module. [ 0.857855] RPC: Registered tcp transport module. [ 0.859052] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.860832] NET: Registered protocol family 44 [ 0.861981] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.866191] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.868759] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.871280] PCI: CLS 0 bytes, default 64 [ 0.874270] Unpacking initramfs... [ 2.787100] debug: unmapping init [mem 0xffff8b5abcc54000-0xffff8b5abffbffff] [ 2.794947] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.796607] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.800970] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.399172] Initialise system trusted keyrings [ 3.400957] Key type blacklist registered [ 3.402896] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.414992] zbud: loaded [ 3.418600] *** VALIDATE nfs *** [ 3.420114] *** VALIDATE nfs4 *** [ 3.422097] pstore: using deflate compression [ 3.427751] Platform Keyring initialized [ 3.582831] NET: Registered protocol family 38 [ 3.585272] Key type asymmetric registered [ 3.586672] Asymmetric key parser 'x509' registered [ 3.589213] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.594157] io scheduler mq-deadline registered [ 3.596409] io scheduler kyber registered [ 3.598936] io scheduler bfq registered [ 3.601317] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.605260] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.609197] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.612348] ACPI: Power Button [PWRF] [ 3.624174] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.634538] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.655835] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.669674] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.697511] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.740733] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.774548] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.782677] Non-volatile memory driver v1.3 [ 3.785135] Linux agpgart interface v0.103 [ 3.847797] virtio_blk virtio1: [vda] 134712 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.852121] vda: detected capacity change from 0 to 68972544 [ 3.874942] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.877664] vdb: detected capacity change from 0 to 1073741824 [ 3.892889] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.895702] vdc: detected capacity change from 0 to 2621440000 [ 3.916389] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.919837] vdd: detected capacity change from 0 to 2621440000 [ 3.938391] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.941899] vde: detected capacity change from 0 to 4294967296 [ 4.011200] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.019732] vdf: detected capacity change from 0 to 4294967296 [ 4.038760] libphy: Fixed MDIO Bus: probed [ 4.061620] usbcore: registered new interface driver usbserial_generic [ 4.066156] usbserial: USB Serial support registered for generic [ 4.069232] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.078972] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.081979] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.085595] mousedev: PS/2 mouse device common for all mice [ 4.093968] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.099172] rtc_cmos 00:05: RTC can wake from S4 [ 4.102089] rtc_cmos 00:05: registered as rtc0 [ 4.103824] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.109696] intel_pstate: CPU model not supported [ 4.112565] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.115579] hid: raw HID events driver (C) Jiri Kosina [ 4.367047] hpet1: lost 16 rtc interrupts [ 4.371931] usbcore: registered new interface driver usbhid [ 4.373756] usbhid: USB HID core driver [ 4.375069] drop_monitor: Initializing network drop monitor service [ 4.376047] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.381503] Initializing XFRM netlink socket [ 4.390703] NET: Registered protocol family 10 [ 4.396590] Segment Routing with IPv6 [ 4.398185] NET: Registered protocol family 17 [ 4.400742] mpls_gso: MPLS GSO support [ 4.407076] RAS: Correctable Errors collector initialized. [ 4.409419] AVX version of gcm_enc/dec engaged. [ 4.416876] AES CTR mode by8 optimization enabled [ 4.514128] sched_clock: Marking stable (4514108687, 0)->(5434242078, -920133391) [ 4.516958] registered taskstats version 1 [ 4.519350] Loading compiled-in X.509 certificates [ 4.521671] zswap: loaded using pool lzo/zbud [ 4.554761] Key type big_key registered [ 4.570195] Key type encrypted registered [ 4.571962] ima: No TPM chip found, activating TPM-bypass! [ 4.574972] ima: Allocated hash algorithm: sha1 [ 4.576842] ima: No architecture policies found [ 4.579353] evm: Initialising EVM extended attributes: [ 4.581534] evm: security.selinux [ 4.582923] evm: security.ima [ 4.584458] evm: security.capability [ 4.585474] evm: HMAC attrs: 0x1 [ 4.587921] rtc_cmos 00:05: setting system clock to 2026-03-02 16:38:21 UTC (1772469501) [ 4.593738] debug: unmapping init [mem 0xffffffffa2003000-0xffffffffa21fffff] [ 4.597597] debug: unmapping init [mem 0xffffffffa0d82000-0xffffffffa1058fff] [ 4.607061] Write protecting the kernel read-only data: 28672k [ 4.610254] debug: unmapping init [mem 0xffffffff9f403000-0xffffffff9f5fffff] [ 4.613334] debug: unmapping init [mem 0xffffffff9fd14000-0xffffffff9fdfffff] [ 4.654363] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.663729] systemd[1]: Detected virtualization kvm. [ 4.665914] systemd[1]: Detected architecture x86-64. [ 4.668300] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.698256] systemd[1]: No hostname configured. [ 4.700355] systemd[1]: Set hostname to . [ 4.702357] random: systemd: uninitialized urandom read (16 bytes read) [ 4.705117] systemd[1]: Initializing machine ID from random generator. [ 4.985313] random: systemd: uninitialized urandom read (16 bytes read) [ 4.989652] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.996173] random: systemd: uninitialized urandom read (16 bytes read) [ 4.999605] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.008976] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.564755] device-mapper: uevent: version 1.0.3 [ 6.567503] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.976804] random: fast init done [ 8.125124] virtio_net virtio0 ens2: renamed from eth0 [ 10.893224] scsi host0: ata_piix [ 10.954201] scsi host1: ata_piix [ 10.958197] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.965838] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.603099] random: crng init done [ 14.607524] random: 7 urandom warning(s) missed due to ratelimiting [ 17.310456] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.562076] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.449283] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.031340] SELinux: Disabled at runtime. [ 24.334024] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.347929] systemd[1]: Detected virtualization kvm. [ 24.351364] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.379692] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.383598] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.394931] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.401426] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.405542] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.412771] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.437544] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. Mounting POSIX M[ 25.540356] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS essage Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 27.347894] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 28.799132] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.881875] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.709700] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.808439] EDAC sbridge: Ver: 1.1.2 [ 32.756686] Key type dns_resolver registered [ 33.220352] NFS: Registering the id_resolver key type [ 33.222751] Key type id_resolver registered [ 33.224777] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg423-server login: [ 108.409696] libcfs: loading out-of-tree module taints kernel. [ 108.469588] Key type ._llcrypt registered [ 108.477640] Key type .llcrypt registered [ 108.615528] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_hostid [ 126.787881] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 128.474941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.483966] alg: No test for adler32 (adler32-zlib) [ 129.942549] Lustre: Lustre: Build Version: 2.17.50_194_gae5720b [ 131.083756] LNet: Added LNI 192.168.204.123@tcp [8/256/0/180] [ 132.855357] Key type lgssc registered [ 134.681988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 146.028162] hrtimer: interrupt took 6013737 ns [ 150.992304] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 197.375888] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing load_modules_local [ 212.681913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 212.727718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.204308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 214.251771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 214.390969] Lustre: lustre-MDT0000: new disk, initializing [ 214.592608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.656802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 219.977235] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.809629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.977174] Lustre: 6516:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 236.053818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 236.061515] Lustre: Skipped 1 previous similar message [ 236.161348] Lustre: lustre-MDT0001: new disk, initializing [ 236.243672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.308399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 236.329139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 241.440554] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 246.800454] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 257.970585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 258.442381] Lustre: lustre-OST0000: new disk, initializing [ 258.445144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 258.522345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.039250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 259.055228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 259.168636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 265.857627] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.893620] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 283.074957] Lustre: lustre-OST0001: new disk, initializing [ 283.090937] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 283.205753] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 290.137118] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 291.426216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 291.435756] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 291.619079] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 303.043123] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 310.417852] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 324.241501] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing check_logdir /tmp/testlogs/ [ 330.729662] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing yml_node [ 335.116282] Lustre: DEBUG MARKER: Client: 2.17.50.194 [ 337.750885] Lustre: DEBUG MARKER: MDS: 2.17.50.194 [ 340.890520] Lustre: DEBUG MARKER: OSS: 2.17.50.194 [ 343.069632] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Mon Mar 2 11:43:58 EST 2026 [ 363.337702] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 365.292529] Lustre: DEBUG MARKER: === replay-single: start setup 11:44:20 (1772469860) === [ 369.872351] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing check_config_client /mnt/lustre [ 392.479965] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 396.681559] Lustre: 13190:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 401.141801] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 407.526258] Lustre: DEBUG MARKER: === replay-single: finish setup 11:45:03 (1772469903) === [ 409.562869] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 11:45:04 (1772469904) [ 418.037672] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 420.004256] Lustre: Failing over lustre-MDT0000 [ 420.352922] Lustre: server umount lustre-MDT0000 complete [ 420.833925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 420.840130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.943622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.963615] Lustre: Skipped 2 previous similar messages [ 425.877544] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 425.894804] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 430.048413] LustreError: 12058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.069880] LustreError: 12058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 435.170854] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.209649] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 440.289791] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.292756] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772469921/real 1772469921] req@ffff8b5b3a3d4700 x1858569111727232/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772469937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 440.304887] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 440.346741] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 442.678327] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 442.681522] LDISKFS-fs (dm-0): recovery complete [ 442.703580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.408492] LustreError: 7859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.417646] LustreError: 7859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 450.852955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.478174] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 455.318367] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 456.175266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 456.289160] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 463.448440] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 465.307229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 474.124408] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 11:46:09 (1772469969) [ 476.833054] Lustre: Failing over lustre-OST0000 [ 476.911704] Lustre: server umount lustre-OST0000 complete [ 477.132791] LustreError: 8457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.158110] LustreError: 8457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 477.177811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 477.193934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 479.199769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 479.205547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.471388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 495.795134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.304963] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 497.449540] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 497.449540] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 497.470502] Lustre: Skipped 3 previous similar messages [ 502.030766] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 512.421210] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 514.539809] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 526.150551] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 11:47:01 (1772470021) [ 535.370669] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 537.938464] Lustre: Failing over lustre-MDT0000 [ 538.501799] Lustre: server umount lustre-MDT0000 complete [ 538.603533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 538.618346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.642801] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.675744] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 559.078193] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772470039/real 1772470039] req@ffff8b5a045aca80 x1858569111792768/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772470055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 559.103201] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.126370] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 563.130348] LDISKFS-fs (dm-0): recovery complete [ 563.138662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.614105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.950233] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 574.971835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 574.980953] Lustre: Skipped 1 previous similar message [ 575.238982] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 578.032338] Lustre: lustre-MDT0000: Denying connection for new client 830d8d6d-2505-4e7d-8c75-fd5bef9f8555 (at 192.168.204.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 583.579023] Lustre: lustre-MDT0000: Denying connection for new client 830d8d6d-2505-4e7d-8c75-fd5bef9f8555 (at 192.168.204.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 583.623858] Lustre: Skipped 1 previous similar message [ 588.715545] Lustre: lustre-MDT0000: Denying connection for new client 830d8d6d-2505-4e7d-8c75-fd5bef9f8555 (at 192.168.204.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 593.826857] Lustre: lustre-MDT0000: Denying connection for new client 830d8d6d-2505-4e7d-8c75-fd5bef9f8555 (at 192.168.204.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:50 [ 598.933578] Lustre: lustre-MDT0000: Denying connection for new client 830d8d6d-2505-4e7d-8c75-fd5bef9f8555 (at 192.168.204.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:45 [ 609.174700] Lustre: lustre-MDT0000: Denying connection for new client 830d8d6d-2505-4e7d-8c75-fd5bef9f8555 (at 192.168.204.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:35 [ 609.193975] Lustre: Skipped 1 previous similar message [ 629.662906] Lustre: lustre-MDT0000: Denying connection for new client 830d8d6d-2505-4e7d-8c75-fd5bef9f8555 (at 192.168.204.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:14 [ 629.688984] Lustre: Skipped 3 previous similar messages [ 644.500717] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 644.503354] Lustre: 17863:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0dbb8299-fc09-44b8-86c8-63d474c6b981@ [ 644.526355] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 644.606103] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 644.606369] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 644.643784] Lustre: Skipped 2 previous similar messages [ 644.684492] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 644.685288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 655.670945] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 11:49:10 (1772470150) [ 666.630896] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 669.637479] Lustre: Failing over lustre-MDT0000 [ 669.848441] Lustre: server umount lustre-MDT0000 complete [ 670.193056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 670.204367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.222671] Lustre: Skipped 3 previous similar messages [ 670.228717] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.245312] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 688.607438] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772470169/real 1772470169] req@ffff8b5a040f3480 x1858569111852160/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772470185 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 688.643406] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 696.950450] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 696.955211] LDISKFS-fs (dm-0): recovery complete [ 696.963048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.899298] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b3f682680 x1858569111861504/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 699.352113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.425642] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 704.484781] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 704.488261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 704.915884] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 708.190857] Lustre: lustre-MDT0000: Denying connection for new client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 708.215647] Lustre: Skipped 2 previous similar messages [ 774.502676] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 774.508148] Lustre: 19584:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 830d8d6d-2505-4e7d-8c75-fd5bef9f8555@ [ 774.539023] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 774.564297] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 774.570370] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 774.570379] Lustre: Skipped 2 previous similar messages [ 774.610777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 774.611551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 785.308536] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 11:51:20 (1772470280) [ 793.821289] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 795.710897] Lustre: Failing over lustre-MDT0000 [ 796.139063] Lustre: server umount lustre-MDT0000 complete [ 796.646788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.665249] LustreError: 18233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 796.672853] Lustre: Skipped 5 previous similar messages [ 796.722847] LustreError: 18233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 31 previous similar messages [ 813.024161] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772470293/real 1772470293] req@ffff8b5a03144380 x1858569111909760/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772470309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 813.064071] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 821.537884] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 821.546919] LDISKFS-fs (dm-0): recovery complete [ 821.586462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.247440] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c222bc5 [ 822.265507] Lustre: MGC192.168.204.123@tcp: Connection restored to 0@lo (at 0@lo) [ 822.762079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 822.827834] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 826.271942] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 826.479713] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 827.990178] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 828.098021] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 828.098983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 835.216715] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 837.335711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 846.879679] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 11:52:22 (1772470342) [ 856.617836] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 859.209995] Lustre: Failing over lustre-MDT0000 [ 861.512370] Lustre: server umount lustre-MDT0000 complete [ 863.725818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.757436] Lustre: Skipped 3 previous similar messages [ 879.568398] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772470360/real 1772470360] req@ffff8b5a03cf3800 x1858569111947136/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772470376 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 879.599449] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 886.420056] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 886.423563] LDISKFS-fs (dm-0): recovery complete [ 886.436658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.298773] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 891.914288] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 895.474614] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 895.486986] Lustre: Skipped 4 previous similar messages [ 895.641139] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 895.724791] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 895.727715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 895.727984] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 903.988448] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 906.052573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 915.322827] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 11:53:30 (1772470410) [ 923.550859] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 926.095906] Lustre: Failing over lustre-MDT0000 [ 926.196381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 926.518949] Lustre: server umount lustre-MDT0000 complete [ 927.689538] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.722572] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 61 previous similar messages [ 947.681186] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772470428/real 1772470428] req@ffff8b5b2f9aa680 x1858569111984640/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772470444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 947.710352] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 950.209107] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 950.211029] LDISKFS-fs (dm-0): recovery complete [ 950.218230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.281878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.291907] Lustre: Skipped 1 previous similar message [ 958.334541] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 958.360339] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 963.562026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 963.567269] Lustre: Skipped 3 previous similar messages [ 963.722146] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 963.750101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:193) [ 963.751132] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 963.808621] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 971.495455] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 973.137458] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 981.577509] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 11:54:37 (1772470477) [ 989.662187] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 992.163692] Lustre: Failing over lustre-MDT0000 [ 992.447367] Lustre: server umount lustre-MDT0000 complete [ 994.281972] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.293816] Lustre: Skipped 8 previous similar messages [ 1010.661065] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772470491/real 1772470491] req@ffff8b5b3f683800 x1858569112020224/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772470507 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1010.690591] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1013.687000] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1013.689857] LDISKFS-fs (dm-0): recovery complete [ 1013.697764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.899061] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b34f14380 x1858569112029568/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1021.278478] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1022.150422] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1024.936640] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1026.626075] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1026.660689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 1026.661661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1031.750850] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1033.200205] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1041.600712] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 11:55:37 (1772470537) [ 1049.400874] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1051.363135] Lustre: Failing over lustre-MDT0000 [ 1051.569940] Lustre: server umount lustre-MDT0000 complete [ 1052.133111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1068.511582] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772470549/real 1772470549] req@ffff8b5b3a401500 x1858569112055168/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772470565 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1068.534485] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1074.400272] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1074.405897] LDISKFS-fs (dm-0): recovery complete [ 1074.426395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.755909] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b338d4700 x1858569112063616/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1079.040848] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1084.203243] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1084.486373] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1084.488419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1094.268879] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1096.806729] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1106.670449] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 11:56:42 (1772470602) [ 1107.636956] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1107.638357] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1107.639826] LustreError: 6524:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b2f9af800 x1858569092879360/t38654705666(0) o35->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:150/0 lens 392/456 e 0 to 0 dl 1772470615 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1116.792431] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1119.089948] Lustre: Failing over lustre-MDT0000 [ 1119.477810] Lustre: server umount lustre-MDT0000 complete [ 1120.226337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1136.607809] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772470617/real 1772470617] req@ffff8b5b036a6300 x1858569112094720/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772470633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1136.636164] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1142.744944] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1142.752086] LDISKFS-fs (dm-0): recovery complete [ 1142.770283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1146.869432] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c22459c [ 1146.882765] Lustre: MGC192.168.204.123@tcp: Connection restored to 0@lo (at 0@lo) [ 1146.893757] Lustre: Skipped 11 previous similar messages [ 1147.325441] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1147.800585] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1147.812805] Lustre: Skipped 1 previous similar message [ 1152.377275] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1152.559688] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1152.575292] Lustre: Skipped 1 previous similar message [ 1152.587653] Lustre: 6524:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b2f98ca80 x1858569092879360/t38654705666(0) o35->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:195/0 lens 392/456 e 0 to 0 dl 1772470660 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1152.622204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1152.622204] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1160.770201] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1162.928071] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1172.915818] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 11:57:48 (1772470668) [ 1180.697143] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1182.896570] Lustre: Failing over lustre-MDT0000 [ 1183.090048] Lustre: server umount lustre-MDT0000 complete [ 1183.204718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1183.213421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.228288] Lustre: Skipped 8 previous similar messages [ 1188.321295] LustreError: 18228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1188.335899] LustreError: 18228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 144 previous similar messages [ 1207.262569] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1207.265932] LDISKFS-fs (dm-0): recovery complete [ 1207.302946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.348933] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b33acb480 x1858569112139648/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1210.862131] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1215.435847] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1216.129859] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1216.132979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1222.312410] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1223.781574] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1231.867756] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 11:58:47 (1772470727) [ 1240.137124] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1241.114813] Lustre: *** cfs_fail_loc=114, val=0*** [ 1243.864541] Lustre: Failing over lustre-MDT0000 [ 1244.359596] Lustre: server umount lustre-MDT0000 complete [ 1267.692887] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1267.694962] LDISKFS-fs (dm-0): recovery complete [ 1267.719827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1272.293054] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c224f89 [ 1272.605205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1272.615285] Lustre: Skipped 4 previous similar messages [ 1277.287364] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1278.084941] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1278.100203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1284.805796] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1286.701129] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1295.584661] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 11:59:51 (1772470791) [ 1302.684791] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1303.654200] Lustre: *** cfs_fail_loc=128, val=0*** [ 1306.501692] Lustre: Failing over lustre-MDT0000 [ 1307.144087] Lustre: server umount lustre-MDT0000 complete [ 1325.037826] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772470805/real 1772470805] req@ffff8b5a03145880 x1858569112200192/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772470821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1325.088923] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1325.099274] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1325.115640] LustreError: Skipped 2 previous similar messages [ 1331.910512] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1331.916175] LDISKFS-fs (dm-0): recovery complete [ 1331.925556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.267070] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b27550a80 x1858569112209792/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1335.771580] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1335.784198] Lustre: Skipped 1 previous similar message [ 1336.592440] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1336.594492] Lustre: Skipped 2 previous similar messages [ 1340.970909] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1340.980752] Lustre: Skipped 2 previous similar messages [ 1341.016531] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1341.020287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1341.751363] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1350.100465] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1351.836429] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1361.363770] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 12:00:56 (1772470856) [ 1369.516832] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1372.066325] Lustre: Failing over lustre-MDT0000 [ 1372.330431] Lustre: server umount lustre-MDT0000 complete [ 1376.745249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1393.884949] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1393.888635] LDISKFS-fs (dm-0): recovery complete [ 1393.896194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.351790] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c225b52 [ 1407.986848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1407.989936] Lustre: Skipped 18 previous similar messages [ 1408.187365] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1408.320092] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1408.320909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1416.265586] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1417.972610] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1427.313550] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 12:02:03 (1772470923) [ 1435.826990] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1438.249950] Lustre: Failing over lustre-MDT0000 [ 1438.595362] Lustre: server umount lustre-MDT0000 complete [ 1463.161923] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1463.165063] LDISKFS-fs (dm-0): recovery complete [ 1463.182678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1470.438868] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b2f9a8700 x1858569112281984/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1470.844778] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1470.860085] Lustre: Skipped 1 previous similar message [ 1475.179467] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1476.291709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1476.293418] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1483.848526] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1485.453458] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1494.604475] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 12:03:10 (1772470990) [ 1502.584466] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1509.657464] Lustre: Failing over lustre-MDT0000 [ 1509.947464] Lustre: server umount lustre-MDT0000 complete [ 1511.903926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.925707] Lustre: Skipped 20 previous similar messages [ 1531.637515] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1531.640790] LDISKFS-fs (dm-0): recovery complete [ 1531.648760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.367909] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1547.390837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1547.398489] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1550.473678] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1552.489666] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1573.661551] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 12:04:29 (1772471069) [ 1580.661641] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1582.254164] Lustre: Failing over lustre-MDT0000 [ 1582.498526] Lustre: server umount lustre-MDT0000 complete [ 1583.583661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1601.503921] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772471082/real 1772471082] req@ffff8b5a03144380 x1858569112409216/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772471098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1601.542551] Lustre: 3667:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1601.549306] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.570033] LustreError: Skipped 3 previous similar messages [ 1603.866071] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1603.870171] LDISKFS-fs (dm-0): recovery complete [ 1603.885821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1612.696267] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1612.707990] Lustre: Skipped 3 previous similar messages [ 1617.213574] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1617.514503] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1617.529147] Lustre: Skipped 3 previous similar messages [ 1617.579760] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1617.583984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1625.283630] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1626.798111] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1636.807391] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 12:05:32 (1772471132) [ 1644.918079] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1646.992197] Lustre: Failing over lustre-MDT0000 [ 1647.379187] Lustre: server umount lustre-MDT0000 complete [ 1667.459285] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1667.464203] LDISKFS-fs (dm-0): recovery complete [ 1667.472786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1674.209411] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b33a4dc00 x1858569112461312/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1679.995724] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1680.000286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1680.292644] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1687.743437] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1689.216400] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1697.674777] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 12:06:33 (1772471193) [ 1704.482285] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1706.213288] Lustre: Failing over lustre-MDT0000 [ 1706.387723] Lustre: server umount lustre-MDT0000 complete [ 1706.968302] LustreError: 18228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1706.981913] LustreError: 18228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 268 previous similar messages [ 1729.223918] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1729.232594] LDISKFS-fs (dm-0): recovery complete [ 1729.241329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1736.558580] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1736.561814] Lustre: Skipped 3 previous similar messages [ 1740.482466] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1741.958520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1741.959872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1746.736491] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1748.157075] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1757.645822] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 12:07:33 (1772471253) [ 1763.999894] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1765.795884] Lustre: Failing over lustre-MDT0000 [ 1766.003246] Lustre: server umount lustre-MDT0000 complete [ 1786.242499] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1786.246665] LDISKFS-fs (dm-0): recovery complete [ 1786.255393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.372923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.378806] Lustre: Skipped 7 previous similar messages [ 1798.241375] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1799.808959] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1799.809969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1804.201858] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1805.454310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1813.694782] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 12:08:29 (1772471309) [ 1820.197430] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1822.501350] Lustre: Failing over lustre-MDT0000 [ 1822.791605] Lustre: server umount lustre-MDT0000 complete [ 1844.350343] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1844.353481] LDISKFS-fs (dm-0): recovery complete [ 1844.363446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.788071] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1857.657533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1857.660599] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1863.783860] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1865.283957] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1874.293408] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 12:09:29 (1772471369) [ 1882.840545] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1885.708706] Lustre: Failing over lustre-MDT0000 [ 1886.310709] Lustre: server umount lustre-MDT0000 complete [ 1911.478352] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1911.480412] LDISKFS-fs (dm-0): recovery complete [ 1911.489549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1914.847782] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b047e5880 x1858569112596480/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1920.491654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1920.500933] Lustre: Skipped 31 previous similar messages [ 1920.658672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1920.659306] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1920.700382] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1929.546177] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1931.429277] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1941.434329] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 12:10:37 (1772471437) [ 1949.944482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1951.918155] Lustre: Failing over lustre-MDT0000 [ 1952.328436] Lustre: server umount lustre-MDT0000 complete [ 1975.517357] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1975.523344] LDISKFS-fs (dm-0): recovery complete [ 1975.538323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.398302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1989.398929] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1989.477958] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1997.016426] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1998.676461] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2008.431582] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 12:11:44 (1772471504) [ 2016.496695] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2018.526873] Lustre: Failing over lustre-MDT0000 [ 2019.094622] Lustre: server umount lustre-MDT0000 complete [ 2042.433499] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2042.436427] LDISKFS-fs (dm-0): recovery complete [ 2042.450567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.433807] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5a07dae680 x1858569112671488/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2051.592520] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2052.180568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 2052.185404] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 2059.408307] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2061.403263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2070.310244] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 12:12:46 (1772471566) [ 2079.153755] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2081.198104] Lustre: Failing over lustre-MDT0000 [ 2081.468731] Lustre: server umount lustre-MDT0000 complete [ 2082.798386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2082.802509] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2082.808805] LustreError: Skipped 1 previous similar message [ 2082.835780] Lustre: Skipped 37 previous similar messages [ 2105.690892] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2105.696366] LDISKFS-fs (dm-0): recovery complete [ 2105.716572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.131875] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2114.112155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 2114.112586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 2121.421712] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2123.273324] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2132.417836] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 12:13:48 (1772471628) [ 2139.645656] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2141.678569] Lustre: Failing over lustre-MDT0000 [ 2142.009131] Lustre: server umount lustre-MDT0000 complete [ 2160.591233] Lustre: 3668:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772471641/real 1772471641] req@ffff8b5b33d6a300 x1858569112731776/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772471657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2160.617109] Lustre: 3668:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2160.626342] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2160.638068] LustreError: Skipped 8 previous similar messages [ 2164.750299] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2164.755519] LDISKFS-fs (dm-0): recovery complete [ 2164.767025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.867096] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c2332a7 [ 2171.809134] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2171.816943] Lustre: Skipped 8 previous similar messages [ 2175.368827] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2176.545654] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2176.564963] Lustre: Skipped 8 previous similar messages [ 2176.611884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 2176.615720] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:897) [ 2182.550582] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2183.879190] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2191.866505] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 12:14:47 (1772471687) [ 2200.769149] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2202.415408] Lustre: Failing over lustre-MDT0000 [ 2202.527683] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 2202.537172] Lustre: Skipped 3 previous similar messages [ 2202.659530] Lustre: server umount lustre-MDT0000 complete [ 2225.701415] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2225.704707] LDISKFS-fs (dm-0): recovery complete [ 2225.712045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.824083] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5a07b2a300 x1858569112775680/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2233.847248] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 2238.345621] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2239.575979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2239.579164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2246.404590] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2248.157112] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2256.972515] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 12:15:52 (1772471752) [ 2264.391678] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2267.003241] Lustre: Failing over lustre-MDT0000 [ 2267.197711] Lustre: server umount lustre-MDT0000 complete [ 2291.307812] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2291.310497] LDISKFS-fs (dm-0): recovery complete [ 2291.331615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2297.391264] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2297.405732] Lustre: Skipped 8 previous similar messages [ 2302.587316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 2302.589175] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 2304.071257] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2312.475704] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2314.074495] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2322.729819] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 12:16:58 (1772471818) [ 2330.617872] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2332.489496] Lustre: Failing over lustre-MDT0000 [ 2332.845076] Lustre: server umount lustre-MDT0000 complete [ 2333.153654] LustreError: 7859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2333.175112] LustreError: 7859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 334 previous similar messages [ 2356.156080] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2356.159643] LDISKFS-fs (dm-0): recovery complete [ 2356.169973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2359.784939] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c2342ec [ 2365.590852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 2365.591287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:993) [ 2365.600456] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2373.182223] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2376.002455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2385.274916] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 12:18:00 (1772471880) [ 2392.576099] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2394.528745] Lustre: Failing over lustre-MDT0000 [ 2395.004664] Lustre: server umount lustre-MDT0000 complete [ 2419.570709] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2419.575621] LDISKFS-fs (dm-0): recovery complete [ 2419.588654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.086485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2423.095371] Lustre: Skipped 9 previous similar messages [ 2427.977316] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2428.553531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2428.553531] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2436.218353] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2438.180880] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2449.007641] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 12:19:04 (1772471944) [ 2456.845558] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2459.194791] Lustre: Failing over lustre-MDT0000 [ 2459.500703] Lustre: server umount lustre-MDT0000 complete [ 2484.525540] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2484.529420] LDISKFS-fs (dm-0): recovery complete [ 2484.535900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2490.854801] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b3a4ded80 x1858569112924416/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2496.248922] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2496.675660] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2496.679250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2504.445429] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2506.360623] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2516.192375] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:20:11 (1772472011) [ 2524.518671] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2526.459597] Lustre: Failing over lustre-MDT0000 [ 2526.680825] Lustre: server umount lustre-MDT0000 complete [ 2550.213895] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2550.216903] LDISKFS-fs (dm-0): recovery complete [ 2550.225461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.366644] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2559.477809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2559.494478] Lustre: Skipped 41 previous similar messages [ 2559.696915] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2559.697824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2567.887981] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2570.200657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2579.316116] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 12:21:15 (1772472075) [ 2585.260760] Lustre: 71989:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 0013c40d-31fb-4f31-94cd-422205a73a0a at adminstrative request [ 2590.507125] Lustre: Failing over lustre-MDT0000 [ 2590.692256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2591.239862] Lustre: server umount lustre-MDT0000 complete [ 2611.008386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2616.696325] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2616.932841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2616.941068] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2626.480296] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2628.857587] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2635.716433] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2645.726904] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2653.510687] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 12:22:28 (1772472148) [ 2654.596650] Lustre: 73895:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 0013c40d-31fb-4f31-94cd-422205a73a0a at adminstrative request [ 2665.746385] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 12:22:41 (1772472161) [ 2675.757685] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2677.950067] Lustre: Failing over lustre-MDT0000 [ 2678.254364] Lustre: server umount lustre-MDT0000 complete [ 2702.319324] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2702.328530] LDISKFS-fs (dm-0): recovery complete [ 2702.350047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.543319] LustreError: 75202:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2704.565124] LustreError: 75202:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8b5a0170aa00 x1858569113042304/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1772472201 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2704.592135] LustreError: 75202:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2709.854031] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2710.723202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2710.730706] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2716.886211] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2718.158725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2727.988540] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:23:43 (1772472223) [ 2736.759746] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2738.655237] Lustre: Failing over lustre-MDT0000 [ 2738.925154] Lustre: server umount lustre-MDT0000 complete [ 2741.218315] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.226976] Lustre: Skipped 38 previous similar messages [ 2761.671152] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2761.680146] LDISKFS-fs (dm-0): recovery complete [ 2761.704291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.867562] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c23653d [ 2772.498631] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2773.578041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2773.582741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2779.561985] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2781.067575] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2789.803502] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 12:24:45 (1772472285) [ 2798.248469] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2801.023297] Lustre: Failing over lustre-MDT0000 [ 2801.439778] Lustre: server umount lustre-MDT0000 complete [ 2804.194366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2804.205028] LustreError: Skipped 5 previous similar messages [ 2821.029857] Lustre: 3668:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772472301/real 1772472301] req@ffff8b5b04333b80 x1858569113108992/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772472317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2821.065387] Lustre: 3668:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2821.080088] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2821.098506] LustreError: Skipped 9 previous similar messages [ 2825.270887] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2825.274633] LDISKFS-fs (dm-0): recovery complete [ 2825.283645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.260611] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2832.277016] Lustre: Skipped 9 previous similar messages [ 2836.815412] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2837.074624] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2837.092679] Lustre: Skipped 9 previous similar messages [ 2837.172570] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2837.173615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2845.452656] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2847.232412] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2855.659314] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 12:25:51 (1772472351) [ 2863.671507] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2865.718436] Lustre: Failing over lustre-MDT0000 [ 2866.176721] Lustre: server umount lustre-MDT0000 complete [ 2889.555272] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2889.560021] LDISKFS-fs (dm-0): recovery complete [ 2889.570852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.894932] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2899.001329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2899.006309] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2906.027730] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2907.770936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2917.301962] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 12:26:52 (1772472412) [ 2925.601821] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2927.789747] Lustre: Failing over lustre-MDT0000 [ 2928.214588] Lustre: server umount lustre-MDT0000 complete [ 2934.754723] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.778680] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 298 previous similar messages [ 2952.608572] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2952.615058] LDISKFS-fs (dm-0): recovery complete [ 2952.630685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.640585] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2955.643702] Lustre: Skipped 9 previous similar messages [ 2959.723914] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2960.977153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 2960.977366] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 2967.945649] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2969.924235] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2978.472968] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:27:54 (1772472474) [ 2986.238475] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2988.343354] Lustre: Failing over lustre-MDT0000 [ 2988.603253] Lustre: server umount lustre-MDT0000 complete [ 3011.300900] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3011.306426] LDISKFS-fs (dm-0): recovery complete [ 3011.315676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.721213] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b3a3d5c00 x1858569113222400/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3023.737266] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3024.576532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 3024.577222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 3031.798623] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3033.466958] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3042.713815] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:28:58 (1772472538) [ 3050.102231] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3052.009645] Lustre: Failing over lustre-MDT0000 [ 3052.497722] Lustre: server umount lustre-MDT0000 complete [ 3078.099811] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3078.101850] LDISKFS-fs (dm-0): recovery complete [ 3078.110240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3082.107204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3082.126654] Lustre: Skipped 9 previous similar messages [ 3087.554645] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 3087.558351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 3087.734693] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3097.852740] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3101.247937] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3115.271945] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 12:30:08 (1772472608) [ 3124.754933] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3127.583925] Lustre: Failing over lustre-MDT0000 [ 3128.156957] Lustre: server umount lustre-MDT0000 complete [ 3152.443748] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3152.445300] LDISKFS-fs (dm-0): recovery complete [ 3152.451496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.594378] LustreError: 88042:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 3154.608297] LustreError: 88042:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8b5b33afbb80 x1858569113299200/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1772472651 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3154.641012] LustreError: 88042:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 3154.937152] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c238741 [ 3160.108565] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3160.568171] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3160.576463] Lustre: Skipped 37 previous similar messages [ 3160.688468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 3160.689842] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 3168.455845] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3170.258565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3180.163408] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:31:15 (1772472675) [ 3188.229238] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3190.407110] Lustre: Failing over lustre-MDT0000 [ 3190.777680] Lustre: server umount lustre-MDT0000 complete [ 3214.290267] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3214.292325] LDISKFS-fs (dm-0): recovery complete [ 3214.320767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3217.904516] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c238cff [ 3222.679354] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3223.636278] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3223.637385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3230.402727] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3232.048461] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3240.203472] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 12:32:16 (1772472736) [ 3247.521595] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3249.597326] Lustre: Failing over lustre-MDT0000 [ 3249.814141] Lustre: server umount lustre-MDT0000 complete [ 3272.440689] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3272.446348] LDISKFS-fs (dm-0): recovery complete [ 3272.458914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.881419] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c239303 [ 3286.709613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3286.711748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 3287.003737] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3296.220959] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3297.915509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3307.676185] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 12:33:23 (1772472803) [ 3315.952121] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3317.913916] Lustre: Failing over lustre-MDT0000 [ 3320.263595] Lustre: server umount lustre-MDT0000 complete [ 3344.804137] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3344.807245] LDISKFS-fs (dm-0): recovery complete [ 3344.822299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.752371] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 3354.762037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 3354.958027] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3364.141506] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3366.392921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3378.069639] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 12:34:33 (1772472873) [ 3379.370801] Lustre: 94315:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 0013c40d-31fb-4f31-94cd-422205a73a0a at adminstrative request [ 3389.563451] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 12:34:45 (1772472885) [ 3399.122451] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3401.198911] Lustre: Failing over lustre-MDT0000 [ 3401.637710] Lustre: server umount lustre-MDT0000 complete [ 3405.796771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3405.803085] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.805905] LustreError: Skipped 2 previous similar messages [ 3405.820821] Lustre: Skipped 39 previous similar messages [ 3415.104804] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3415.108928] LDISKFS-fs (dm-0): recovery complete [ 3415.121531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3415.530925] Lustre: lustre-MDT0000: Aborting client recovery [ 3415.535326] LustreError: 95599:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3415.547593] Lustre: 95635:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3415.554487] Lustre: 95635:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0013c40d-31fb-4f31-94cd-422205a73a0a@ [ 3415.563674] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3415.570911] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3415.588592] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3415.652753] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1479 to 0x2c0000401:1505) [ 3415.663470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1480 to 0x280000401:1505) [ 3420.647852] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3420.769951] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3442.217202] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 12:35:37 (1772472937) [ 3449.993171] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3451.870303] Lustre: Failing over lustre-MDT0000 [ 3452.276911] Lustre: server umount lustre-MDT0000 complete [ 3468.358490] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3468.361225] LDISKFS-fs (dm-0): recovery complete [ 3468.378461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3468.565244] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3468.577424] LustreError: Skipped 9 previous similar messages [ 3468.702224] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (not set up) [ 3469.044197] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3469.070187] Lustre: lustre-MDT0000: Aborting client recovery [ 3469.072113] LustreError: 97464:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3469.089475] Lustre: 97500:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3469.097055] Lustre: 97500:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3469.109579] Lustre: 97500:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3469.121900] Lustre: 97500:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3469.129861] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3469.137456] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3469.153831] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3469.233797] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1516 to 0x2c0000401:1537) [ 3469.241240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1516 to 0x280000401:1537) [ 3474.428618] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3474.466671] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3488.501671] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3488.503031] Lustre: Skipped 1 previous similar message [ 3497.262824] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 12:36:32 (1772472992) [ 3506.352553] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3508.635840] Lustre: Failing over lustre-MDT0000 [ 3509.074554] Lustre: server umount lustre-MDT0000 complete [ 3525.525459] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3525.527508] LDISKFS-fs (dm-0): recovery complete [ 3525.557292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.515379] Lustre: lustre-MDT0000: Aborting client recovery [ 3526.528769] LustreError: 99322:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3526.547460] Lustre: 99358:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3526.560749] Lustre: 99358:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3526.589045] Lustre: 99358:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3526.605657] Lustre: 99358:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3526.613342] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3526.630727] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3526.653684] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3526.741656] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 3526.773586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 3531.757364] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3532.600158] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3555.243220] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 12:37:30 (1772473050) [ 3556.456636] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3556.464705] LustreError: 18228:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b33a5b100 x1858569093985792/t201863462916(0) o36->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:334/0 lens 512/456 e 0 to 0 dl 1772473064 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3560.422288] Lustre: Failing over lustre-MDT0000 [ 3560.727415] Lustre: server umount lustre-MDT0000 complete [ 3560.919432] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.939464] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 280 previous similar messages [ 3570.066478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.601621] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3570.602272] Lustre: lustre-MDT0000: Aborting client recovery [ 3570.605069] Lustre: Skipped 15 previous similar messages [ 3570.609782] LustreError: 100637:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3570.612616] Lustre: 100674:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3570.615325] Lustre: 100674:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3570.618981] Lustre: 100674:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3570.623753] Lustre: 100674:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3570.627835] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3570.634751] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3570.646781] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3570.703928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 3570.719288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1601) [ 3575.449233] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3575.800379] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3600.031694] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3602.234595] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 12:38:17 (1772473097) [ 3613.025515] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3616.964454] Lustre: Failing over lustre-MDT0000 [ 3617.336448] Lustre: server umount lustre-MDT0000 complete [ 3632.821980] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3632.824068] LDISKFS-fs (dm-0): recovery complete [ 3632.837816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.244292] Lustre: lustre-MDT0000: Aborting client recovery [ 3633.254629] LustreError: 102595:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3633.274112] Lustre: 102630:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3633.281273] Lustre: 102630:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 3633.289893] Lustre: 102630:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3633.298088] Lustre: 102630:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3633.303646] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3633.310441] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3633.327142] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3633.402448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 3633.403403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1633) [ 3638.269664] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3638.323252] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3663.757408] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:39:19 (1772473159) [ 3695.356298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3697.554790] Lustre: Failing over lustre-MDT0000 [ 3697.805964] Lustre: server umount lustre-MDT0000 complete [ 3716.065457] Lustre: 3668:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772473196/real 1772473196] req@ffff8b5a05110e00 x1858569113703808/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772473212 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3716.104584] Lustre: 3668:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 3720.715848] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3720.719181] LDISKFS-fs (dm-0): recovery complete [ 3720.726187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.303882] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b2724ce00 x1858569113713152/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3726.319438] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 42 previous similar messages [ 3726.533492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.538338] Lustre: Skipped 9 previous similar messages [ 3726.748732] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3726.757581] Lustre: Skipped 8 previous similar messages [ 3731.497281] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3732.123098] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 3732.145926] Lustre: Skipped 8 previous similar messages [ 3732.181922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3732.181986] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3740.079791] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3742.404410] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3768.645950] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 12:41:03 (1772473263) [ 3801.109176] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3811.992966] Lustre: Failing over lustre-MDT0000 [ 3812.362596] Lustre: server umount lustre-MDT0000 complete [ 3838.921331] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3838.923655] LDISKFS-fs (dm-0): recovery complete [ 3838.944352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3845.922596] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3846.131244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3846.153184] Lustre: Skipped 41 previous similar messages [ 3850.024778] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3850.029285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3854.859363] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3856.702572] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3876.712366] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 12:42:52 (1772473372) [ 3878.792589] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3879.777970] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3879.785613] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3885.270600] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 12:43:01 (1772473381) [ 3913.608883] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3926.537927] Lustre: Failing over lustre-OST0000 [ 3926.663183] Lustre: server umount lustre-OST0000 complete [ 3951.662170] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3951.668657] LDISKFS-fs (dm-2): recovery complete [ 3951.676614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3957.262878] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4015.641907] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 12:45:10 (1772473510) [ 4023.458956] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4025.715307] Lustre: Failing over lustre-MDT0000 [ 4025.991906] Lustre: server umount lustre-MDT0000 complete [ 4028.896609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4028.909179] Lustre: Skipped 30 previous similar messages [ 4046.849742] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4046.851976] LDISKFS-fs (dm-0): recovery complete [ 4046.865550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.572797] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4060.222266] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 4060.224258] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 4063.337397] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4064.539865] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4076.511308] LustreError: 111809:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4076.516328] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4077.536487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 4081.830061] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 12:46:17 (1772473577) [ 4085.696276] LustreError: 6522:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4090.847173] LustreError: 6522:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4090.858993] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnecting [ 4090.918855] LustreError: 42262:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 4092.477409] LustreError: 18228:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4097.503140] LustreError: 18228:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4097.508756] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnecting [ 4098.984434] LustreError: 12058:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4104.159195] LustreError: 12058:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4104.169296] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnecting [ 4105.760867] LustreError: 6523:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4110.815578] LustreError: 6523:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4112.758987] LustreError: 12058:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4117.983199] LustreError: 12058:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4117.991632] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnecting [ 4117.997782] Lustre: Skipped 1 previous similar message [ 4126.409438] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4126.414108] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4131.807548] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4131.814258] LustreError: 6521:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4138.976327] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnecting [ 4138.991811] Lustre: Skipped 2 previous similar messages [ 4147.241369] LustreError: 18228:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4147.246881] LustreError: 18228:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4152.287170] LustreError: 18228:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4152.297652] LustreError: 18228:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4159.701159] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 12:47:35 (1772473655) [ 4161.211715] LustreError: 6521:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4171.671851] Lustre: lustre-MDT0000: Export ffff8b5b3a0ea000 already connecting from 192.168.204.23@tcp [ 4176.803545] Lustre: lustre-MDT0000: Export ffff8b5b3a0ea000 already connecting from 192.168.204.23@tcp [ 4181.917191] Lustre: lustre-MDT0000: Export ffff8b5b3a0ea000 already connecting from 192.168.204.23@tcp [ 4187.040255] Lustre: lustre-MDT0000: Export ffff8b5b3a0ea000 already connecting from 192.168.204.23@tcp [ 4187.048524] Lustre: Skipped 1 previous similar message [ 4192.155137] Lustre: lustre-MDT0000: Export ffff8b5b3a0ea000 already connecting from 192.168.204.23@tcp [ 4201.263270] LustreError: 6521:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4201.266679] Lustre: 6521:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b5b33e10e00 x1858569096651520/t0(0) o38->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:0/0 lens 520/416 e 0 to 0 dl 1772473678 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4202.390159] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnecting [ 4202.407462] Lustre: Skipped 3 previous similar messages [ 4202.411258] LustreError: 12058:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4226.973439] Lustre: lustre-MDT0000: Export ffff8b5b3a0ea000 already connecting from 192.168.204.23@tcp [ 4226.982468] Lustre: Skipped 1 previous similar message [ 4242.495173] LustreError: 12058:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4242.498978] Lustre: 12058:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b5b2f916300 x1858569096655872/t0(0) o38->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:0/0 lens 520/416 e 0 to 0 dl 1772473719 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4247.446936] LustreError: 18233:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4272.034492] Lustre: lustre-MDT0000: Export ffff8b5b3a0ea000 already connecting from 192.168.204.23@tcp [ 4272.039890] Lustre: Skipped 4 previous similar messages [ 4287.495720] LustreError: 18233:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4287.502882] Lustre: 18233:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b5b04b9f100 x1858569096659456/t0(0) o38->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:0/0 lens 520/416 e 0 to 0 dl 1772473764 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4292.503155] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnecting [ 4292.508101] Lustre: Skipped 1 previous similar message [ 4292.510345] LustreError: 6523:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4317.086890] Lustre: lustre-MDT0000: Export ffff8b5b3a0ea000 already connecting from 192.168.204.23@tcp [ 4317.095581] Lustre: Skipped 4 previous similar messages [ 4332.519124] LustreError: 6523:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4332.523920] Lustre: 6523:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b5b3f681c00 x1858569096663040/t0(0) o38->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:0/0 lens 520/416 e 0 to 0 dl 1772473809 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4337.542351] LustreError: 12058:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4377.543592] LustreError: 12058:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4377.548604] Lustre: 12058:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8b5a07dac380 x1858569096666624/t0(0) o38->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:0/0 lens 520/416 e 0 to 0 dl 1772473854 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4384.399926] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 12:51:20 (1772473880) [ 4389.432369] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4392.227732] Lustre: Failing over lustre-MDT0000 [ 4392.378422] Lustre: server umount lustre-MDT0000 complete [ 4392.871249] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.894637] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 139 previous similar messages [ 4392.928831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4392.935190] LustreError: Skipped 5 previous similar messages [ 4401.172290] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4401.175783] LDISKFS-fs (dm-0): recovery complete [ 4401.181273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4401.248877] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4401.254437] LustreError: Skipped 6 previous similar messages [ 4401.372778] Lustre: *** cfs_fail_loc=712, val=0*** [ 4401.376665] LustreError: 42259:0:(service.c:1390:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8b5b04a70000 x1858569114423296/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4401.407260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4401.411397] Lustre: Skipped 3 previous similar messages [ 4401.429444] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4401.429535] Lustre: lustre-MDT0000: Aborting client recovery [ 4401.431514] Lustre: Skipped 9 previous similar messages [ 4401.432895] LustreError: 116551:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4401.445369] Lustre: 116588:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4401.449985] Lustre: 116588:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 4401.454294] Lustre: 116588:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4401.459519] Lustre: 116588:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4401.463965] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4401.469197] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4401.481878] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4401.538369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 4401.543286] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 4404.325702] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4406.754870] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4417.933406] Lustre: Failing over lustre-MDT0000 [ 4418.225117] Lustre: server umount lustre-MDT0000 complete [ 4434.334688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.878906] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4434.887714] Lustre: Skipped 3 previous similar messages [ 4437.999456] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4440.066546] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4440.075049] Lustre: Skipped 3 previous similar messages [ 4440.114220] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 4440.118246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 4443.914875] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4445.090846] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4450.583905] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 12:52:26 (1772473946) [ 4450.686285] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnecting [ 4450.696684] Lustre: Skipped 2 previous similar messages [ 4456.528296] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 12:52:32 (1772473952) [ 4457.200448] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4457.205415] LustreError: 6530:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b2f864e00 x1858569096726656/t0(0) o700->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:480/0 lens 264/248 e 0 to 0 dl 1772473965 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4476.054456] Lustre: Failing over lustre-MDT0000 [ 4476.249688] Lustre: server umount lustre-MDT0000 complete [ 4492.180487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.754385] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4497.899357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4497.909030] Lustre: Skipped 19 previous similar messages [ 4497.952602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 4497.952770] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 4501.574402] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4502.847613] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4511.010608] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:53:27 (1772474007) [ 4513.047396] Lustre: Failing over lustre-OST0000 [ 4513.133946] Lustre: server umount lustre-OST0000 complete [ 4529.085179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4533.164814] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4538.841792] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4540.350460] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4609.200773] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 12:55:05 (1772474105) [ 4615.757149] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4617.808922] Lustre: Failing over lustre-MDT0000 [ 4618.062395] Lustre: server umount lustre-MDT0000 complete [ 4636.113993] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4636.116070] LDISKFS-fs (dm-0): recovery complete [ 4636.122285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.085427] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4642.127151] Lustre: *** cfs_fail_loc=216, val=0*** [ 4642.129376] LustreError: 122380:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 4642.136415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4643.170766] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4708.153897] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 12:56:44 (1772474204) [ 4709.159810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4709.166392] Lustre: Skipped 20 previous similar messages [ 4709.169465] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4709.175598] Lustre: Skipped 2 previous similar messages [ 4718.400464] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 12:56:54 (1772474214) [ 4720.109133] Lustre: Failing over lustre-MDT0000 [ 4720.263532] Lustre: server umount lustre-MDT0000 complete [ 4735.458188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4738.184340] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4741.100731] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4741.105383] LustreError: 123954:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b33a63b80 x1858569096850432/t0(0) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:49/0 lens 328/344 e 0 to 0 dl 1772474289 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4797.845924] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnected, waiting for 2 clients in recovery for 1:05 [ 4797.866176] Lustre: 123954:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4797.916663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 4797.917905] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 4800.754939] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4801.638655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4807.266367] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 12:58:23 (1772474303) [ 4808.939377] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4813.515397] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4814.664827] Lustre: Failing over lustre-MDT0000 [ 4814.847645] Lustre: server umount lustre-MDT0000 complete [ 4832.150557] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4832.152726] LDISKFS-fs (dm-0): recovery complete [ 4832.157886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4835.125490] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4837.894724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 4837.898910] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 4840.082405] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4840.962680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4847.346623] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 12:59:03 (1772474343) [ 4848.326181] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4854.858098] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4856.286548] Lustre: Failing over lustre-MDT0000 [ 4856.430946] Lustre: server umount lustre-MDT0000 complete [ 4874.193911] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772474355/real 1772474355] req@ffff8b5b33b3b800 x1858569114686720/t0(0) o400->MGC192.168.204.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1772474371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4874.208558] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4874.238482] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4874.240201] LDISKFS-fs (dm-0): recovery complete [ 4874.244635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4884.451939] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629c282f2b [ 4887.611211] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4890.127032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3169) [ 4890.133343] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 4892.649430] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4893.818356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4898.987895] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 12:59:55 (1772474395) [ 4899.656749] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4905.667345] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4906.908032] Lustre: Failing over lustre-MDT0000 [ 4907.078458] Lustre: server umount lustre-MDT0000 complete [ 4924.510718] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4924.512692] LDISKFS-fs (dm-0): recovery complete [ 4924.523919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4926.730414] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4930.055183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3201) [ 4930.061127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 4936.510444] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 13:00:32 (1772474432) [ 4938.310139] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4938.312892] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4938.317310] LustreError: 6524:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b046ab100 x1858569096904576/t257698037777(0) o35->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:206/0 lens 392/456 e 0 to 0 dl 1772474446 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4940.069321] Lustre: Failing over lustre-MDT0000 [ 4940.222716] Lustre: server umount lustre-MDT0000 complete [ 4955.115446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.590954] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4960.784061] Lustre: 6525:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b3a4ded80 x1858569096904576/t257698037777(0) o35->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:228/0 lens 392/456 e 0 to 0 dl 1772474468 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4960.785398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 4960.785460] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3201) [ 4962.993597] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4964.107310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4969.245346] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 13:01:05 (1772474465) [ 4969.943573] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4969.946339] LustreError: 18228:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b33ab5c00 x1858569096918016/t261993005072(0) o36->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:281/0 lens 504/448 e 0 to 0 dl 1772474521 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4975.815696] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4976.907213] Lustre: Failing over lustre-MDT0000 [ 4977.053906] Lustre: server umount lustre-MDT0000 complete [ 4993.434441] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4993.444429] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 177 previous similar messages [ 4994.635748] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4994.638190] LDISKFS-fs (dm-0): recovery complete [ 4994.643974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4997.171298] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5000.198873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 5000.198889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3233) [ 5000.199743] Lustre: 6521:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b2fb00700 x1858569096918016/t261993005072(0) o36->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:312/0 lens 504/2880 e 0 to 0 dl 1772474552 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5002.242905] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5003.224381] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5008.354795] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 13:01:44 (1772474504) [ 5008.881824] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5008.883742] LustreError: 18233:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b3a4dea00 x1858569096932864/t266287972368(0) o36->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:320/0 lens 504/448 e 0 to 0 dl 1772474560 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5010.291368] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5014.236142] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5015.281514] Lustre: Failing over lustre-MDT0000 [ 5015.411116] Lustre: server umount lustre-MDT0000 complete [ 5015.524113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5015.526318] LustreError: Skipped 7 previous similar messages [ 5032.177494] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5032.185247] LDISKFS-fs (dm-0): recovery complete [ 5032.197903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5032.270490] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5032.275396] LustreError: Skipped 9 previous similar messages [ 5032.426239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5032.430065] Lustre: Skipped 10 previous similar messages [ 5032.461093] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5032.469491] Lustre: Skipped 12 previous similar messages [ 5035.022231] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5035.413158] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5035.418935] Lustre: Skipped 9 previous similar messages [ 5037.561280] Lustre: 6522:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b046aaa00 x1858569096932864/t266287972368(0) o36->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:349/0 lens 504/2880 e 0 to 0 dl 1772474589 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5037.569804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 5037.571716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297) [ 5037.584623] Lustre: 6522:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5043.615068] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 13:02:20 (1772474540) [ 5044.152953] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5044.155220] Lustre: Skipped 1 previous similar message [ 5044.156991] LustreError: 18228:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b33b7e300 x1858569096946176/t270582939664(0) o36->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:356/0 lens 504/448 e 0 to 0 dl 1772474596 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5044.165175] LustreError: 18228:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 5045.505836] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5045.508619] Lustre: Skipped 1 previous similar message [ 5049.109910] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5050.065652] Lustre: Failing over lustre-MDT0000 [ 5050.193834] Lustre: server umount lustre-MDT0000 complete [ 5066.050490] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5066.053480] LDISKFS-fs (dm-0): recovery complete [ 5066.059092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.134717] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (not set up) [ 5067.898668] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5071.334532] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5071.337922] Lustre: Skipped 10 previous similar messages [ 5071.351076] Lustre: 12058:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b037d3480 x1858569096946176/t270582939664(0) o36->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:383/0 lens 504/2880 e 0 to 0 dl 1772474623 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5071.359758] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 5071.360058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3329) [ 5076.405761] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 13:02:52 (1772474572) [ 5076.988134] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5078.379820] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5078.381102] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5078.383456] LustreError: 6524:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b27754850 x1858569096959104/t274877906960(0) o35->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:346/0 lens 392/456 e 0 to 0 dl 1772474586 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5083.103114] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5084.071844] Lustre: Failing over lustre-MDT0000 [ 5084.211669] Lustre: server umount lustre-MDT0000 complete [ 5100.862521] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5100.865973] LDISKFS-fs (dm-0): recovery complete [ 5100.874065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5102.929167] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5106.145681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5106.149742] Lustre: Skipped 44 previous similar messages [ 5106.166748] Lustre: 6525:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b33a4d500 x1858569096959104/t274877906960(0) o35->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:374/0 lens 392/456 e 0 to 0 dl 1772474614 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5106.174953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3361) [ 5106.176353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 5112.115877] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 13:03:28 (1772474608) [ 5112.673156] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 5112.676732] LustreError: 18228:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b33ab7480 x1858569096969600/t279172874255(0) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:424/0 lens 664/608 e 0 to 0 dl 1772474664 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5172.634807] Lustre: 18228:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b33a61880 x1858569096969600/t279172874255(0) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:484/0 lens 664/3488 e 0 to 0 dl 1772474724 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5176.190357] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 13:04:32 (1772474672) [ 5180.260395] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5181.149074] Lustre: Failing over lustre-MDT0000 [ 5181.288040] Lustre: server umount lustre-MDT0000 complete [ 5197.298370] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5197.299789] LDISKFS-fs (dm-0): recovery complete [ 5197.309026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5199.423712] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5202.955875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 5202.956313] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3361) [ 5204.808252] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5205.642170] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5221.122410] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 13:05:17 (1772474717) [ 5225.786852] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5226.905638] Lustre: Failing over lustre-MDT0000 [ 5227.112822] Lustre: server umount lustre-MDT0000 complete [ 5242.941105] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5242.943193] LDISKFS-fs (dm-0): recovery complete [ 5242.948315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5244.715335] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5248.513587] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3393) [ 5248.514574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 5250.102756] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5250.850398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5253.864655] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5260.615740] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 13:05:57 (1772474757) [ 5285.920256] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5286.954030] Lustre: Failing over lustre-MDT0000 [ 5287.240492] Lustre: server umount lustre-MDT0000 complete [ 5303.515116] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5303.517213] LDISKFS-fs (dm-0): recovery complete [ 5303.521657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.553951] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5309.086123] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 5309.091604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 5310.754662] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5311.596290] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5343.465859] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 13:07:19 (1772474839) [ 5348.815275] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5350.113456] Lustre: Failing over lustre-MDT0000 [ 5350.314492] Lustre: server umount lustre-MDT0000 complete [ 5350.370481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5350.381338] Lustre: Skipped 49 previous similar messages [ 5367.040207] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5367.042375] LDISKFS-fs (dm-0): recovery complete [ 5367.048351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.387288] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5372.448098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 5372.449202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 5374.208225] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5375.045487] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5379.271744] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5380.056587] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5383.136515] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 13:07:59 (1772474879) [ 5389.549435] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5449.124604] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnecting [ 5449.129940] Lustre: Skipped 2 previous similar messages [ 5450.388409] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5450.393286] Lustre: Skipped 1 previous similar message [ 5450.395807] LustreError: 114310:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b3a4dd880 x1858569099763840/t296352743435(0) o36->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:7/0 lens 66040/440 e 0 to 0 dl 1772475002 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5510.555877] Lustre: 18228:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b0a5f7850 x1858569099763840/t296352743435(0) o36->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:67/0 lens 66040/440 e 0 to 0 dl 1772475062 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5515.567780] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5516.423844] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 13:10:12 (1772475012) [ 5523.180160] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5524.513432] Lustre: Failing over lustre-MDT0000 [ 5524.658722] Lustre: server umount lustre-MDT0000 complete [ 5540.794597] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5540.796321] LDISKFS-fs (dm-0): recovery complete [ 5540.801444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5543.049708] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5546.896427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 5546.896900] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 5548.430123] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5549.134246] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5554.127285] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 13:10:50 (1772475050) [ 5565.098454] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5571.122339] Lustre: Failing over lustre-OST0000 [ 5571.180028] Lustre: server umount lustre-OST0000 complete [ 5587.154106] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5587.155931] LDISKFS-fs (dm-2): recovery complete [ 5587.163747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5589.634702] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5601.742077] Lustre: Failing over lustre-OST0000 [ 5601.797947] Lustre: server umount lustre-OST0000 complete [ 5602.711982] LustreError: 8457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5602.719162] LustreError: 8457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 138 previous similar messages [ 5616.360459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5619.255019] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5623.053938] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5623.815361] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5659.412844] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 13:12:35 (1772475155) [ 5660.877273] Lustre: Failing over lustre-MDT0000 [ 5661.083343] Lustre: server umount lustre-MDT0000 complete [ 5662.179680] LustreError: 40409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772475159 with bad export cookie 10037793917214957032 [ 5675.489954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5675.597475] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5675.600993] LustreError: Skipped 7 previous similar messages [ 5675.752894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5675.761223] Lustre: Skipped 9 previous similar messages [ 5675.793879] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5675.795782] Lustre: Skipped 9 previous similar messages [ 5678.073809] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5679.511052] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5679.517182] Lustre: Skipped 9 previous similar messages [ 5681.135758] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5681.143463] Lustre: Skipped 8 previous similar messages [ 5681.164351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 5681.167761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 5690.233468] Lustre: Failing over lustre-MDT0000 [ 5690.427687] Lustre: server umount lustre-MDT0000 complete [ 5691.360582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5691.364431] LustreError: Skipped 7 previous similar messages [ 5704.152669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5706.017210] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5709.795256] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5709.803342] Lustre: Skipped 31 previous similar messages [ 5709.851122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 5709.851189] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 5711.506431] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5712.222720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5716.363864] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 13:13:32 (1772475212) [ 5728.060521] Lustre: Failing over lustre-OST0000 [ 5728.112861] Lustre: server umount lustre-OST0000 complete [ 5741.876373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5744.340449] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5748.062233] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5748.791486] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5753.355431] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 13:14:09 (1772475249) [ 5754.141770] Lustre: Failing over lustre-MDT0000 [ 5754.332415] Lustre: server umount lustre-MDT0000 complete [ 5757.992273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5758.058947] Lustre: *** cfs_fail_loc=605, val=0*** [ 5758.060328] LustreError: 157865:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0773860 failed: rc = -95 [ 5758.064384] LustreError: 157865:0:(obd_config.c:880:class_setup()) setup MGS failed (-95) [ 5758.067466] LustreError: 157865:0:(obd_mount.c:217:lustre_start_simple()) MGS setup error -95 [ 5758.070105] LustreError: 157865:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 5758.072757] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5758.075308] LustreError: 157865:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-MDT0000 [ 5758.081406] Lustre: server umount lustre-MDT0000 complete [ 5758.083232] LustreError: 157865:0:(super25.c:186:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5761.792366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5763.518897] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5767.117430] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 13:14:23 (1772475263) [ 5767.161960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 5767.162062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 5770.289757] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5771.563250] Lustre: Failing over lustre-MDT0000 [ 5771.675634] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 5771.784345] Lustre: server umount lustre-MDT0000 complete [ 5786.475095] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5786.476450] LDISKFS-fs (dm-0): recovery complete [ 5786.479505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.036993] Lustre: *** cfs_fail_loc=707, val=0*** [ 5788.118740] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5847.472577] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 5847.493498] Lustre: 160030:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5847.504663] Lustre: 160030:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 5847.725701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 5847.728261] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 5849.851534] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5850.627772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5855.688306] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 13:15:52 (1772475352) [ 5881.184461] LustreError: 6522:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b5b33b38380 x1858569100647296/t0(0) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:394/0 lens 664/0 e 0 to 0 dl 1772475389 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5881.199596] LustreError: 6522:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 5892.231146] LustreError: 6522:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5892.249788] LustreError: 6524:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b5b33a60a80 x1858569100648576/t0(0) o35->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:405/0 lens 392/0 e 0 to 0 dl 1772475400 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5898.209801] LustreError: 42268:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b5b046a8e00 x1858569116160640/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:411/0 lens 544/0 e 0 to 0 dl 1772475406 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 5898.223549] LustreError: 42268:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 48 previous similar messages [ 5905.298794] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 13:16:41 (1772475401) [ 5930.301708] LustreError: 41204:0:(tgt_handler.c:2783:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 5941.327155] LustreError: 41204:0:(tgt_handler.c:2783:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 5945.468524] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 13:17:22 (1772475442) [ 5968.998630] LustreError: 114310:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b5b04332300 x1858569100672768/t0(0) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:481/0 lens 576/0 e 0 to 0 dl 1772475476 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5969.006956] LustreError: 114310:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 5974.103138] LustreError: 114310:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5974.110713] LustreError: 18228:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b5b046ab100 x1858569100673664/t0(0) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:488/0 lens 664/0 e 0 to 0 dl 1772475483 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5975.321547] LustreError: 6523:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 10000ms [ 5985.423118] LustreError: 6523:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5985.432355] LustreError: 114310:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b5b2f9a9f80 x1858569100695424/t0(0) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:538/0 lens 664/0 e 0 to 0 dl 1772475533 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5985.445541] LustreError: 114310:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 5999.018411] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 13:18:15 (1772475495) [ 6085.702180] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 13:19:42 (1772475582) [ 6109.278388] LustreError: 18228:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b5b2724c380 x1858569100752384/t0(0) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:622/0 lens 576/0 e 0 to 0 dl 1772475617 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6109.288840] LustreError: 18228:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 6109.292090] LustreError: 18228:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6109.711122] LustreError: 18228:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6125.678270] LustreError: 12058:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6125.682129] LustreError: 12058:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 6126.111124] LustreError: 12058:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6126.114081] LustreError: 12058:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 6141.650331] LustreError: 6522:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8b5b33acaa00 x1858569100772352/t0(0) o36->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:654/0 lens 504/0 e 0 to 0 dl 1772475649 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 6141.661418] LustreError: 6522:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 80 previous similar messages [ 6154.367907] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 13:20:50 (1772475650) [ 6179.405240] Lustre: DEBUG MARKER: phase 2 [ 6182.919851] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 13:21:19 (1772475679) [ 6255.207183] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 13:22:31 (1772475751) [ 6255.772492] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6256.403407] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 13:22:33 (1772475753) [ 6258.272648] Lustre: DEBUG MARKER: Started rundbench load pid=134483 ... [ 6262.302099] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6263.902665] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6264.703516] Lustre: Failing over lustre-MDT0000 [ 6264.985741] Lustre: server umount lustre-MDT0000 complete [ 6265.494938] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6265.499657] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 67 previous similar messages [ 6267.872713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6267.881142] Lustre: Skipped 32 previous similar messages [ 6280.259952] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 6280.261655] LDISKFS-fs (dm-0): recovery complete [ 6280.266847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6280.311579] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6280.315779] LustreError: Skipped 3 previous similar messages [ 6280.395183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6280.398813] Lustre: Skipped 4 previous similar messages [ 6280.415324] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6280.418888] Lustre: Skipped 4 previous similar messages [ 6281.850276] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6283.669304] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6283.672732] Lustre: Skipped 4 previous similar messages [ 6285.808570] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6285.812012] Lustre: Skipped 4 previous similar messages [ 6285.828199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) [ 6285.828447] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) [ 6287.069125] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6287.738433] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6292.985940] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6294.582607] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6295.285174] Lustre: Failing over lustre-MDT0001 [ 6295.425479] Lustre: server umount lustre-MDT0001 complete [ 6309.943157] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 6309.946965] LDISKFS-fs (dm-1): recovery complete [ 6309.954029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6311.359684] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6315.489117] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6315.492196] Lustre: Skipped 17 previous similar messages [ 6315.890535] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:682 to 0x280000400:705) [ 6315.890547] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:664 to 0x2c0000400:705) [ 6317.283425] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6317.934312] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6323.563601] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6325.175318] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 6325.901166] Lustre: Failing over lustre-MDT0000 [ 6326.129083] Lustre: server umount lustre-MDT0000 complete [ 6330.847261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6330.850223] LustreError: Skipped 4 previous similar messages [ 6340.748463] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6340.750145] LDISKFS-fs (dm-0): recovery complete [ 6340.755260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6342.277967] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6346.236096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) [ 6346.236111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) [ 6347.433515] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6348.024364] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6352.938484] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6354.460886] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 6355.195414] Lustre: Failing over lustre-MDT0001 [ 6355.200495] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 6355.445668] Lustre: server umount lustre-MDT0001 complete [ 6369.395567] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6369.396910] LDISKFS-fs (dm-1): recovery complete [ 6369.400048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.688979] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6375.124588] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:997 to 0x2c0000400:1025) [ 6375.124594] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:997 to 0x280000400:1025) [ 6376.319992] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6376.880511] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6404.304646] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 13:25:00 (1772475900) [ 6527.174184] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6537.888828] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 6538.549180] Lustre: Failing over lustre-MDT0001 [ 6538.719969] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6538.722183] Lustre: Skipped 2 previous similar messages [ 6538.782740] Lustre: server umount lustre-MDT0001 complete [ 6552.743929] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6552.745124] LDISKFS-fs (dm-1): recovery complete [ 6552.749682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.039958] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6560.021302] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2521 to 0x280000400:2561) [ 6560.021480] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2520 to 0x2c0000400:2561) [ 6561.177715] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6561.688306] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6685.739519] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6696.420560] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 6697.019939] Lustre: Failing over lustre-MDT0001 [ 6697.025747] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 6697.249686] Lustre: server umount lustre-MDT0001 complete [ 6711.244511] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6711.245683] LDISKFS-fs (dm-1): recovery complete [ 6711.249653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.512601] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6718.423257] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3950 to 0x2c0000400:3969) [ 6718.423316] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3949 to 0x280000400:3969) [ 6719.573408] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6720.094189] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6736.357634] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 13:30:33 (1772476233) [ 6859.153453] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6869.840544] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 6870.448490] Lustre: Failing over lustre-MDT0001 [ 6870.460429] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 6872.031427] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6872.036472] Lustre: Skipped 16 previous similar messages [ 6876.228176] Lustre: server umount lustre-MDT0001 complete [ 6876.565820] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.571798] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 69 previous similar messages [ 6890.092113] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6890.093135] LDISKFS-fs (dm-1): recovery complete [ 6890.097716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6890.203785] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6890.214663] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6890.217275] Lustre: Skipped 5 previous similar messages [ 6891.451669] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6891.743675] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6891.746025] Lustre: Skipped 5 previous similar messages [ 6896.667238] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6896.669549] Lustre: Skipped 5 previous similar messages [ 6896.684546] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4494 to 0x280000400:4513) [ 6896.684873] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4493 to 0x2c0000400:4513) [ 6897.819532] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6898.333973] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7022.140969] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7032.912572] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 7033.579412] Lustre: Failing over lustre-MDT0000 [ 7033.584042] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 7033.585995] Lustre: Skipped 5 previous similar messages [ 7033.823976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7033.826406] LustreError: Skipped 3 previous similar messages [ 7033.951897] Lustre: server umount lustre-MDT0000 complete [ 7048.253651] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7048.255235] LDISKFS-fs (dm-0): recovery complete [ 7048.259658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7048.297914] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7048.302033] LustreError: Skipped 1 previous similar message [ 7048.388856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7048.392873] Lustre: Skipped 5 previous similar messages [ 7049.748199] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7053.793905] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7053.797171] Lustre: Skipped 18 previous similar messages [ 7054.920614] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8798 to 0x2c0000401:8833) [ 7054.920624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8862 to 0x280000401:8897) [ 7056.129959] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7056.684344] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7059.953551] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 13:35:56 (1772476556) [ 7183.011761] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7193.685597] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 7194.282258] Lustre: Failing over lustre-MDT0000 [ 7194.306594] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 7194.309913] Lustre: Skipped 4 previous similar messages [ 7194.440316] Lustre: server umount lustre-MDT0000 complete [ 7208.490680] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7208.492116] LDISKFS-fs (dm-0): recovery complete [ 7208.494906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7209.866995] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7217.326448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10952 to 0x280000401:10977) [ 7217.333468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10887 to 0x2c0000401:10913) [ 7218.616923] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7219.208564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7343.353108] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7354.154780] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 7354.818926] Lustre: Failing over lustre-MDT0001 [ 7355.072466] Lustre: server umount lustre-MDT0001 complete [ 7369.217757] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7369.219098] LDISKFS-fs (dm-1): recovery complete [ 7369.222596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7370.601644] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7374.839268] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4988 to 0x280000400:5025) [ 7374.839310] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4989 to 0x2c0000400:5025) [ 7376.160297] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7376.777895] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7380.387805] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 13:41:17 (1772476877) [ 7386.352981] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7387.881790] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7388.519055] Lustre: Failing over lustre-OST0000 [ 7388.561468] Lustre: server umount lustre-OST0000 complete [ 7402.698224] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7402.700611] LDISKFS-fs (dm-2): recovery complete [ 7402.704278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7404.640205] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7407.828357] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7408.454878] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7417.344417] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7418.829812] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 7419.469082] Lustre: Failing over lustre-OST0000 [ 7419.511230] Lustre: server umount lustre-OST0000 complete [ 7433.479164] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7433.480571] LDISKFS-fs (dm-2): recovery complete [ 7433.483448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7435.457063] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7438.705678] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7439.348562] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7444.981929] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 13:42:21 (1772476941) [ 7567.952346] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7570.304554] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7581.051022] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 7581.719811] Lustre: Failing over lustre-MDT0000 [ 7581.732082] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.23@tcp (stopping) [ 7581.903211] Lustre: server umount lustre-MDT0000 complete [ 7582.175943] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7582.176334] LustreError: 139276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7582.179691] Lustre: Skipped 20 previous similar messages [ 7582.185809] LustreError: 139276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 66 previous similar messages [ 7583.101845] LustreError: 40409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477080 with bad export cookie 10037793917221478029 [ 7583.102966] Lustre: Failing over lustre-MDT0001 [ 7583.106648] LustreError: 40409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7583.254502] Lustre: server umount lustre-MDT0001 complete [ 7598.147688] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7598.150905] LDISKFS-fs (dm-1): recovery complete [ 7598.156859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7598.203454] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7598.205314] LDISKFS-fs (dm-0): recovery complete [ 7598.208701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7598.249498] LustreError: 190268:0:(llog.c:1643:llog_backup()) MGC192.168.204.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7598.253094] Lustre: 190268:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7603.679193] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772477084/real 1772477084] req@ffff8b5a077e5880 x1858569160107904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772477100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7603.692186] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7608.799484] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5a0b66f800 x1858569160110848/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7608.809229] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 7608.913021] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7608.916747] Lustre: Skipped 5 previous similar messages [ 7609.005208] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7609.009440] Lustre: Skipped 5 previous similar messages [ 7610.408374] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7610.549722] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7615.079576] Lustre: lustre-MDT0001: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 7615.083952] Lustre: Skipped 5 previous similar messages [ 7615.099319] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4988 to 0x280000400:5057) [ 7615.099414] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4989 to 0x2c0000400:5057) [ 7620.353028] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12994 to 0x2c0000401:13025) [ 7620.353226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13058 to 0x280000401:13089) [ 7621.715387] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7622.308393] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7622.892115] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7747.098882] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7749.675605] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7760.447989] Lustre: DEBUG MARKER: fail mds2 mds1 2 times [ 7761.126317] Lustre: Failing over lustre-MDT0001 [ 7761.129987] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 7761.132992] LustreError: Skipped 5 previous similar messages [ 7761.135193] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7761.277763] Lustre: server umount lustre-MDT0001 complete [ 7762.512958] Lustre: Failing over lustre-MDT0000 [ 7768.196354] Lustre: server umount lustre-MDT0000 complete [ 7783.269317] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7783.272535] LDISKFS-fs (dm-1): recovery complete [ 7783.278603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7783.320302] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7783.322577] LDISKFS-fs (dm-0): recovery complete [ 7783.326770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7783.369699] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7783.374461] LustreError: Skipped 2 previous similar messages [ 7783.376833] LustreError: 193520:0:(llog.c:1643:llog_backup()) MGC192.168.204.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7783.377074] LustreError: 193518:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGC192.168.204.123@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff8b5a0c2b8200) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7783.379929] Lustre: 193520:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7783.467564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7783.471111] Lustre: Skipped 6 previous similar messages [ 7784.992423] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7785.094613] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7789.025800] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7789.027607] Lustre: Skipped 20 previous similar messages [ 7789.621278] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4989 to 0x2c0000400:5089) [ 7789.621380] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4988 to 0x280000400:5089) [ 7795.226775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13058 to 0x280000401:13121) [ 7795.230856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12994 to 0x2c0000401:13057) [ 7796.412087] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7796.973525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7797.526695] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7800.564544] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 13:48:17 (1772477297) [ 7802.828361] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7803.663109] Lustre: Failing over lustre-MDT0000 [ 7803.809165] Lustre: server umount lustre-MDT0000 complete [ 7817.767350] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7817.768575] LDISKFS-fs (dm-0): recovery complete [ 7817.771263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.080471] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7819.670441] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7835.041960] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7835.054298] Lustre: 195308:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b33b39500 x1858569191021568/t335007470382(335007470382) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:82/0 lens 520/3488 e 0 to 0 dl 1772477342 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7835.094707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13153) [ 7835.094747] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12994 to 0x2c0000401:13089) [ 7836.565464] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7837.129542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7840.714711] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 13:48:57 (1772477337) [ 7843.291384] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7844.187560] Lustre: Failing over lustre-MDT0000 [ 7844.320196] Lustre: server umount lustre-MDT0000 complete [ 7858.265582] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7858.266889] LDISKFS-fs (dm-0): recovery complete [ 7858.269886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7859.512022] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7860.637638] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7860.639180] LustreError: 193529:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b3bda1c00 x1858569191021568/t335007470382(335007470382) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:108/0 lens 520/664 e 0 to 0 dl 1772477368 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7875.995713] Lustre: lustre-MDT0000: Client 0013c40d-31fb-4f31-94cd-422205a73a0a (at 192.168.204.23@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7876.000518] Lustre: 193527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b3f682300 x1858569191021568/t335007470382(335007470382) o101->0013c40d-31fb-4f31-94cd-422205a73a0a@192.168.204.23@tcp:123/0 lens 520/3488 e 0 to 0 dl 1772477383 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7876.029405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13185) [ 7876.029468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13121) [ 7877.339994] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7877.913656] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7881.399187] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 13:49:38 (1772477378) [ 7883.124207] Lustre: Failing over lustre-OST0000 [ 7883.170081] Lustre: server umount lustre-OST0000 complete [ 7884.408826] Lustre: Failing over lustre-MDT0000 [ 7884.505842] Lustre: server umount lustre-MDT0000 complete [ 7897.395621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7898.853325] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7899.591708] Lustre: lustre-MDT0000: Denying connection for new client 9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6 (at 192.168.204.23@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7899.596918] Lustre: Skipped 13 previous similar messages [ 7902.708049] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13091 to 0x2c0000401:13153) [ 7906.723963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7908.384417] Lustre: lustre-OST0000: Denying connection for new client 9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6 (at 192.168.204.23@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 7908.390838] Lustre: Skipped 1 previous similar message [ 7908.668201] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7908.722801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13217) [ 7912.818325] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 13:50:09 (1772477409) [ 7913.170170] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7913.171522] LustreError: 194282:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5a0c930000 x1858569173468544/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:161/0 lens 1312/4320 e 0 to 0 dl 1772477421 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7915.650387] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7916.283336] Lustre: Failing over lustre-MDT0000 [ 7916.510402] Lustre: server umount lustre-MDT0000 complete [ 7929.311212] Lustre: 193553:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772477410/real 1772477410] req@ffff8b5a0c932a00 x1858569173468544/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1772477426 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7929.323504] Lustre: 193553:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 7930.521412] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7930.523427] LDISKFS-fs (dm-0): recovery complete [ 7930.526821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.044579] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7935.998109] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13185) [ 7935.998180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13249) [ 7937.212714] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7937.737649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7941.388915] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 13:50:38 (1772477438) [ 7944.310982] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7946.894211] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7947.559596] Lustre: Failing over lustre-MDT0001 [ 7947.564909] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 7947.568090] Lustre: Skipped 4 previous similar messages [ 7952.968699] Lustre: server umount lustre-MDT0001 complete [ 7967.131596] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7967.132911] LDISKFS-fs (dm-1): recovery complete [ 7967.137149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7968.547671] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7972.333789] Lustre: 193529:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b3a3d7800 x1858569205631360/t38654708482(0) o36->9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6@192.168.204.23@tcp:241/0 lens 560/2880 e 0 to 0 dl 1772477501 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7972.340509] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5100 to 0x280000400:5121) [ 7972.340756] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5100 to 0x2c0000400:5121) [ 7973.572764] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7974.126581] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7977.730357] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 13:51:14 (1772477474) [ 7978.117836] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7978.119951] Lustre: Skipped 1 previous similar message [ 7978.121773] LustreError: 193533:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b33a62300 x1858569173531392/t360777252897(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:226/0 lens 2520/4320 e 0 to 0 dl 1772477486 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7978.128923] LustreError: 193533:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 7980.614708] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7983.140938] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7983.818993] Lustre: Failing over lustre-MDT0000 [ 7984.144140] Lustre: server umount lustre-MDT0000 complete [ 7997.862456] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7997.863552] LDISKFS-fs (dm-0): recovery complete [ 7997.866203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.178472] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8003.087421] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13217) [ 8003.087440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13281) [ 8004.174191] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8004.713674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8006.650346] Lustre: Failing over lustre-MDT0001 [ 8006.728311] Lustre: server umount lustre-MDT0001 complete [ 8020.614749] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8020.617088] LDISKFS-fs (dm-1): recovery complete [ 8020.622549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8021.964800] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8026.100192] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5132 to 0x280000400:5153) [ 8026.100220] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5132 to 0x2c0000400:5153) [ 8027.233279] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8027.748705] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8031.164224] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 13:52:07 (1772477527) [ 8036.972431] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8039.517600] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8040.201356] Lustre: Failing over lustre-MDT0000 [ 8040.412403] Lustre: server umount lustre-MDT0000 complete [ 8041.575840] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477538 with bad export cookie 10037793917224591286 [ 8041.576730] Lustre: Failing over lustre-MDT0001 [ 8041.582814] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8047.071637] LustreError: 209224:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b5b34f61180 x1858569173593728/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8047.078380] LustreError: 209224:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8047.183134] Lustre: server umount lustre-MDT0001 complete [ 8062.053699] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8062.056078] LDISKFS-fs (dm-0): recovery complete [ 8062.059639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8062.065202] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8062.066525] LDISKFS-fs (dm-1): recovery complete [ 8062.072174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8067.039819] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b0443f100 x1858569173594496/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8068.670248] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8068.776852] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8072.615993] Lustre: 211084:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b3bda1880 x1858569205698176/t47244640324(0) o36->9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6@192.168.204.23@tcp:345/0 lens 560/2880 e 0 to 0 dl 1772477605 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8072.622679] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5164 to 0x280000400:5185) [ 8072.622892] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5164 to 0x2c0000400:5185) [ 8072.650835] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13249) [ 8072.650881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13313) [ 8074.169963] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8074.799131] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8075.402239] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8079.096347] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 13:52:55 (1772477575) [ 8079.457495] LustreError: 210342:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b037d3480 x1858569205730688/t51539607620(0) o36->9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6@192.168.204.23@tcp:366/0 lens 560/448 e 0 to 0 dl 1772477626 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8079.468784] LustreError: 210342:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 8084.956421] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8085.616111] Lustre: Failing over lustre-MDT0000 [ 8085.955855] Lustre: server umount lustre-MDT0000 complete [ 8099.865342] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8099.866588] LDISKFS-fs (dm-0): recovery complete [ 8099.869875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8101.235043] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8105.463464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13345) [ 8105.463502] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13281) [ 8106.679450] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8107.227780] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8134.048552] Lustre: lustre-MDT0001: Client 9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6 (at 192.168.204.23@tcp) reconnecting [ 8134.051058] Lustre: Skipped 1 previous similar message [ 8134.053668] Lustre: 210342:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b3b913480 x1858569205730688/t51539607620(0) o36->9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6@192.168.204.23@tcp:420/0 lens 560/2880 e 0 to 0 dl 1772477680 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8136.427218] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 13:53:53 (1772477633) [ 8136.876274] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8136.878351] Lustre: Skipped 2 previous similar messages [ 8139.443922] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8140.032817] Lustre: Failing over lustre-MDT0001 [ 8140.120967] Lustre: server umount lustre-MDT0001 complete [ 8154.094205] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8154.095702] LDISKFS-fs (dm-1): recovery complete [ 8154.100536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8155.378031] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8159.729634] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5206 to 0x2c0000400:5249) [ 8159.729638] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5206 to 0x280000400:5249) [ 8160.818908] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8161.330609] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8164.822518] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 13:54:21 (1772477661) [ 8170.704746] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8173.219906] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8173.824219] Lustre: Failing over lustre-MDT0000 [ 8174.085774] Lustre: server umount lustre-MDT0000 complete [ 8185.311669] LustreError: 211084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8185.316650] LustreError: 211084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 171 previous similar messages [ 8188.067504] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8188.068850] LDISKFS-fs (dm-0): recovery complete [ 8188.072218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.491535] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8193.535481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13377) [ 8193.535506] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13313) [ 8194.681538] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8195.302132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8197.267921] Lustre: Failing over lustre-MDT0001 [ 8197.367433] Lustre: server umount lustre-MDT0001 complete [ 8198.623965] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8198.628189] Lustre: Skipped 52 previous similar messages [ 8211.300483] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8211.302250] LDISKFS-fs (dm-1): recovery complete [ 8211.306835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8211.419658] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8211.423081] Lustre: Skipped 16 previous similar messages [ 8212.514995] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8212.518991] Lustre: Skipped 16 previous similar messages [ 8212.685137] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8216.550280] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8216.554723] Lustre: Skipped 16 previous similar messages [ 8216.570195] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5260 to 0x280000400:5281) [ 8216.570552] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5260 to 0x2c0000400:5281) [ 8217.792631] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8218.379384] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8222.076678] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 13:55:18 (1772477718) [ 8222.450470] LustreError: 211087:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5b042b2680 x1858569205830144/t60129542212(0) o36->9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6@192.168.204.23@tcp:470/0 lens 560/448 e 0 to 0 dl 1772477730 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8222.460260] LustreError: 211087:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 2 previous similar messages [ 8227.849648] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8230.251942] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8230.873215] Lustre: Failing over lustre-MDT0000 [ 8231.160589] Lustre: server umount lustre-MDT0000 complete [ 8232.376611] LustreError: 145555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477729 with bad export cookie 10037793917224605013 [ 8232.377307] Lustre: Failing over lustre-MDT0001 [ 8232.380181] LustreError: 145555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8232.479283] Lustre: server umount lustre-MDT0001 complete [ 8247.148120] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8247.150568] LDISKFS-fs (dm-0): recovery complete [ 8247.153590] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8247.154867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.155464] LDISKFS-fs (dm-1): recovery complete [ 8247.163328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.242961] LustreError: 220945:0:(llog.c:1643:llog_backup()) MGC192.168.204.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8247.245762] Lustre: 220945:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8252.383204] Lustre: 3668:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772477733/real 1772477733] req@ffff8b5b33ab4380 x1858569173764736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772477749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8252.394276] Lustre: 3668:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 8257.503291] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b33805880 x1858569173767168/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8258.898103] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8259.042308] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8263.735195] Lustre: 220978:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b34f61500 x1858569205830144/t60129542212(0) o36->9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6@192.168.204.23@tcp:511/0 lens 560/2880 e 0 to 0 dl 1772477771 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8263.741293] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5292 to 0x2c0000400:5313) [ 8263.741300] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5292 to 0x280000400:5313) [ 8263.748152] Lustre: 220978:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages [ 8263.780719] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13345) [ 8263.780728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13409) [ 8265.005427] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8265.542330] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8266.068304] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8269.519069] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 13:56:06 (1772477766) [ 8272.361510] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8272.956321] Lustre: Failing over lustre-MDT0001 [ 8272.964373] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.23@tcp (stopping) [ 8272.966357] Lustre: Skipped 12 previous similar messages [ 8279.090936] Lustre: server umount lustre-MDT0001 complete [ 8293.197401] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8293.198801] LDISKFS-fs (dm-1): recovery complete [ 8293.202191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8294.548419] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8298.485459] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5324 to 0x280000400:5345) [ 8298.485467] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5324 to 0x2c0000400:5345) [ 8299.633828] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8300.187142] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8303.783118] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 13:56:40 (1772477800) [ 8306.473053] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8307.078041] Lustre: Failing over lustre-MDT0000 [ 8307.291627] Lustre: server umount lustre-MDT0000 complete [ 8321.460055] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8321.461364] LDISKFS-fs (dm-0): recovery complete [ 8321.464761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8322.887441] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8326.659554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13441) [ 8326.659554] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13377) [ 8327.821546] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8328.413130] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8331.880918] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 13:57:08 (1772477828) [ 8334.796562] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8337.372350] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8338.031657] Lustre: Failing over lustre-MDT0000 [ 8338.280490] Lustre: server umount lustre-MDT0000 complete [ 8352.188771] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8352.190098] LDISKFS-fs (dm-0): recovery complete [ 8352.193359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8353.548322] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8357.374448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13473) [ 8357.374453] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13409) [ 8358.570361] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8359.148060] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8361.152587] Lustre: Failing over lustre-MDT0001 [ 8361.241259] Lustre: server umount lustre-MDT0001 complete [ 8362.463848] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8362.466036] LustreError: Skipped 15 previous similar messages [ 8375.117735] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8375.119593] LDISKFS-fs (dm-1): recovery complete [ 8375.123527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.425309] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8380.404608] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5324 to 0x2c0000400:5377) [ 8380.404626] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5324 to 0x280000400:5377) [ 8381.477580] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8382.037626] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8385.407158] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 13:58:02 (1772477882) [ 8388.074539] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8390.419224] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8391.021174] Lustre: Failing over lustre-MDT0000 [ 8391.229725] Lustre: server umount lustre-MDT0000 complete [ 8392.485895] LustreError: 9474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772477889 with bad export cookie 10037793917224615450 [ 8392.487579] Lustre: Failing over lustre-MDT0001 [ 8392.488225] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8392.488242] LustreError: Skipped 11 previous similar messages [ 8392.491862] LustreError: 9474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8392.495099] LustreError: 230442:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0x1f:0x0].0xf7117594 (ffff8b5b02a21900) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8397.792097] LustreError: 230442:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8b5b3a292d80 x1858569173907712/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8397.797548] LustreError: 230442:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8397.885809] Lustre: server umount lustre-MDT0001 complete [ 8412.675499] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8412.678128] LDISKFS-fs (dm-0): recovery complete [ 8412.679709] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8412.682330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.682585] LDISKFS-fs (dm-1): recovery complete [ 8412.692029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8417.248312] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629cc007c8 [ 8417.250762] Lustre: MGC192.168.204.123@tcp: Connection restored to 0@lo (at 0@lo) [ 8417.253025] Lustre: Skipped 72 previous similar messages [ 8417.360298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8417.362983] Lustre: Skipped 21 previous similar messages [ 8418.833424] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8418.959475] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8423.566425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13505) [ 8423.566444] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13441) [ 8423.569862] Lustre: 231559:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8b5b343f6a00 x1858569205903744/t73014444034(0) o36->9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6@192.168.204.23@tcp:697/0 lens 496/2888 e 0 to 0 dl 1772477957 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8423.569874] Lustre: 231559:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8423.577873] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5324 to 0x2c0000400:5409) [ 8423.578198] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5324 to 0x280000400:5409) [ 8424.784363] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8425.300765] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8425.803311] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8429.022235] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 13:58:45 (1772477925) [ 8429.363159] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8429.364402] Lustre: Skipped 6 previous similar messages [ 8431.948851] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8432.542139] Lustre: Failing over lustre-MDT0000 [ 8432.685843] Lustre: server umount lustre-MDT0000 complete [ 8445.845204] Lustre: lustre-MDT0001: Client 9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6 (at 192.168.204.23@tcp) reconnecting [ 8445.847883] Lustre: Skipped 1 previous similar message [ 8446.623141] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8446.624403] LDISKFS-fs (dm-0): recovery complete [ 8446.628508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8448.177988] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8452.091420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13537) [ 8452.091478] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13473) [ 8453.288837] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8453.832267] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8457.390666] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 13:59:14 (1772477954) [ 8460.601752] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8461.200199] Lustre: Failing over lustre-MDT0001 [ 8461.287808] Lustre: server umount lustre-MDT0001 complete [ 8475.130514] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8475.132539] LDISKFS-fs (dm-1): recovery complete [ 8475.136621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8476.410850] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8480.760551] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5324 to 0x2c0000400:5441) [ 8480.760574] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5324 to 0x280000400:5441) [ 8481.807307] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8482.294455] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8485.524595] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 13:59:42 (1772477982) [ 8485.877078] LustreError: 232304:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff8b5a08a4b800 x1858569205940864/t81604378626(0) o36->9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6@192.168.204.23@tcp:733/0 lens 496/456 e 0 to 0 dl 1772477993 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8485.883322] LustreError: 232304:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 6 previous similar messages [ 8488.194847] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8490.514858] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8491.101842] Lustre: Failing over lustre-MDT0000 [ 8491.253850] Lustre: server umount lustre-MDT0000 complete [ 8505.035051] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8505.036239] LDISKFS-fs (dm-0): recovery complete [ 8505.039912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8506.372452] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8510.477679] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13505) [ 8510.477699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13569) [ 8511.605671] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8512.142870] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8514.061332] Lustre: Failing over lustre-MDT0001 [ 8514.157866] Lustre: server umount lustre-MDT0001 complete [ 8527.832198] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8527.833462] LDISKFS-fs (dm-1): recovery complete [ 8527.838329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8529.106558] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8532.980448] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5324 to 0x2c0000400:5473) [ 8532.980535] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5324 to 0x280000400:5473) [ 8534.155106] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8534.667621] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8538.142969] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 14:00:34 (1772478034) [ 8540.971525] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8543.572333] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8544.226983] Lustre: Failing over lustre-MDT0000 [ 8544.483889] Lustre: server umount lustre-MDT0000 complete [ 8545.700634] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478042 with bad export cookie 10037793917224623010 [ 8545.702101] Lustre: Failing over lustre-MDT0001 [ 8545.703928] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8545.801484] Lustre: server umount lustre-MDT0001 complete [ 8560.549056] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8560.551474] LDISKFS-fs (dm-0): recovery complete [ 8560.551640] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8560.554940] LDISKFS-fs (dm-1): recovery complete [ 8560.558597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.561501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.649094] LustreError: 242183:0:(llog.c:1643:llog_backup()) MGC192.168.204.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8560.653337] Lustre: 242183:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8564.191119] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1772478045/real 1772478045] req@ffff8b5a023c0000 x1858569174040576/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1772478061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8564.199213] Lustre: 3665:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 8570.335715] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8b5b3a68f800 x1858569174043904/t0(0) o250->MGC192.168.204.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8570.342497] LustreError: 3664:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 8571.841513] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8571.940939] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8576.638224] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5324 to 0x2c0000400:5505) [ 8576.638234] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5324 to 0x280000400:5505) [ 8576.694074] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13155 to 0x2c0000401:13537) [ 8576.694109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13601) [ 8577.972796] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8578.497802] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8578.991067] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8582.546844] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 14:01:19 (1772478079) [ 8583.140953] Lustre: 243436:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 9834dd8b-3fe9-4c35-b4d7-076b81e9e7e6 at adminstrative request [ 8587.893607] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 14:01:24 (1772478084) [ 8589.890847] Lustre: Failing over lustre-MDT0000 [ 8590.174277] Lustre: server umount lustre-MDT0000 complete [ 8602.866685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8604.283356] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8608.272995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13589 to 0x2c0000401:13633) [ 8608.272996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13652 to 0x280000401:13697) [ 8609.510379] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8610.135928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8613.470387] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 14:01:50 (1772478110) [ 8616.965595] Lustre: Failing over lustre-OST0000 [ 8617.026254] Lustre: server umount lustre-OST0000 complete [ 8629.620847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8631.480151] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8634.429370] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8634.917781] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8638.503695] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 14:02:15 (1772478135) [ 8639.908957] Lustre: Failing over lustre-MDT0000 [ 8640.082762] Lustre: server umount lustre-MDT0000 complete [ 8642.552685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8643.951692] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8644.652612] Lustre: lustre-MDT0000: Denying connection for new client c1c77f1e-22c5-49f1-967e-4f6b84a0bae1 (at 192.168.204.23@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 8648.180334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13652 to 0x280000401:13729) [ 8648.180354] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13589 to 0x2c0000401:13665) [ 8652.152895] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 14:02:28 (1772478148) [ 8654.902299] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8655.741901] Lustre: Failing over lustre-OST0000 [ 8655.773574] Lustre: server umount lustre-OST0000 complete [ 8669.834392] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8669.836194] LDISKFS-fs (dm-2): recovery complete [ 8669.839326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8671.691210] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8674.733379] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8675.247540] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8678.669272] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 14:02:55 (1772478175) [ 8681.348352] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8683.207200] Lustre: Failing over lustre-OST0000 [ 8683.244460] Lustre: server umount lustre-OST0000 complete [ 8697.088066] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8697.089371] LDISKFS-fs (dm-2): recovery complete [ 8697.092488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8698.623146] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.23@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:13731 extent [0-4194303]: client csum 4bc3792, server csum 5aa849b6 [ 8698.985798] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8702.007453] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8702.566107] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8705.876906] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 14:03:22 (1772478202) [ 8708.272365] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8710.708653] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8712.581097] Lustre: Failing over lustre-MDT0000 [ 8712.830924] Lustre: server umount lustre-MDT0000 complete [ 8714.044908] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478210 with bad export cookie 10037793917224649841 [ 8714.047134] Lustre: Failing over lustre-OST0000 [ 8714.050296] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8714.086658] Lustre: server umount lustre-OST0000 complete [ 8728.332794] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8728.334829] LDISKFS-fs (dm-0): recovery complete [ 8728.338282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.272627] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8b4d68629cc09389 [ 8739.577769] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8743.497398] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13589 to 0x2c0000401:13697) [ 8753.634818] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8753.636012] LDISKFS-fs (dm-2): recovery complete [ 8753.639019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8755.414414] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8760.788442] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 14:04:17 (1772478257) [ 8765.424586] Lustre: Failing over lustre-OST0000 [ 8765.489338] Lustre: server umount lustre-OST0000 complete [ 8766.663146] Lustre: Failing over lustre-MDT0000 [ 8766.899328] Lustre: server umount lustre-MDT0000 complete [ 8779.558174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8780.847133] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8784.883740] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13589 to 0x2c0000401:13729) [ 8786.324982] LustreError: 42262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8786.329676] LustreError: 42262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 273 previous similar messages [ 8790.885751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8792.761685] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8793.487657] Lustre: lustre-OST0000: Denying connection for new client 07dd3627-4f21-47fc-8ab6-dbf936330522 (at 192.168.204.23@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 8793.492765] Lustre: Skipped 1 previous similar message [ 8798.613208] Lustre: lustre-OST0000: Denying connection for new client 07dd3627-4f21-47fc-8ab6-dbf936330522 (at 192.168.204.23@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 8798.618526] Lustre: Skipped 1 previous similar message [ 8808.853154] Lustre: lustre-OST0000: Denying connection for new client 07dd3627-4f21-47fc-8ab6-dbf936330522 (at 192.168.204.23@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 8808.857731] Lustre: Skipped 1 previous similar message [ 8829.333103] Lustre: lustre-OST0000: Denying connection for new client 07dd3627-4f21-47fc-8ab6-dbf936330522 (at 192.168.204.23@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:33 [ 8829.338240] Lustre: Skipped 3 previous similar messages [ 8862.500134] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 8862.502220] Lustre: 256483:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client c1c77f1e-22c5-49f1-967e-4f6b84a0bae1@ [ 8862.506260] Lustre: 256483:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 8862.508513] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 8862.514790] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 8862.517769] Lustre: Skipped 22 previous similar messages [ 8862.518820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13780 to 0x280000401:13801) [ 8866.114518] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 8870.504702] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 8872.716281] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 14:06:09 (1772478369) [ 8874.137126] Lustre: Failing over lustre-OST0001 [ 8874.189712] Lustre: server umount lustre-OST0001 complete [ 8877.023627] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.026969] Lustre: Skipped 67 previous similar messages [ 8887.357532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8887.418293] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [ 8887.420512] Lustre: Skipped 23 previous similar messages [ 8888.618432] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8888.620671] Lustre: Skipped 23 previous similar messages [ 8889.309799] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8893.218603] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 14:06:29 (1772478389) [ 8894.449761] Lustre: Failing over lustre-OST0000 [ 8894.487950] Lustre: server umount lustre-OST0000 complete [ 8907.100898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8908.878785] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8909.239870] LustreError: 259460:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 8909.242812] LustreError: 259460:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 77 previous similar messages [ 8915.935160] Lustre: *** cfs_fail_loc=715, val=40*** [ 8925.548466] Lustre: lustre-OST0000: Client 07dd3627-4f21-47fc-8ab6-dbf936330522 (at 192.168.204.23@tcp) reconnected, waiting for 3 clients in recovery for 1:23 [ 8931.807131] Lustre: *** cfs_fail_loc=715, val=40*** [ 8931.808223] Lustre: Skipped 2 previous similar messages [ 8940.948994] Lustre: lustre-OST0000: Client 07dd3627-4f21-47fc-8ab6-dbf936330522 (at 192.168.204.23@tcp) reconnected, waiting for 3 clients in recovery for 1:08 [ 8940.953014] Lustre: Skipped 2 previous similar messages [ 8942.047688] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:07 [ 8942.052422] Lustre: Skipped 1 previous similar message [ 8947.167216] Lustre: *** cfs_fail_loc=715, val=40*** [ 8947.171151] Lustre: Skipped 2 previous similar messages [ 8949.287061] LustreError: 259460:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 8949.289303] LustreError: 259460:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 77 previous similar messages [ 8950.634072] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8951.144786] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8954.441732] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 14:07:31 (1772478451) [ 8955.796301] Lustre: Failing over lustre-MDT0000 [ 8956.048752] Lustre: server umount lustre-MDT0000 complete [ 8968.854489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8970.179096] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8974.307826] LustreError: 260904:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 8977.887128] Lustre: *** cfs_fail_loc=715, val=80*** [ 8977.889446] Lustre: Skipped 2 previous similar messages [ 8987.029114] Lustre: lustre-MDT0000: Client 07dd3627-4f21-47fc-8ab6-dbf936330522 (at 192.168.204.23@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 8990.687543] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 8993.247116] Lustre: *** cfs_fail_loc=715, val=80*** [ 8993.248523] Lustre: Skipped 1 previous similar message [ 9003.412675] Lustre: lustre-MDT0000: Client 07dd3627-4f21-47fc-8ab6-dbf936330522 (at 192.168.204.23@tcp) reconnected, waiting for 2 clients in recovery for 1:11 [ 9007.071438] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9009.631135] Lustre: *** cfs_fail_loc=715, val=80*** [ 9009.632924] Lustre: Skipped 1 previous similar message [ 9019.796920] Lustre: lustre-MDT0000: Client 07dd3627-4f21-47fc-8ab6-dbf936330522 (at 192.168.204.23@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 9022.431512] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9026.015122] Lustre: *** cfs_fail_loc=715, val=80*** [ 9026.016317] Lustre: Skipped 1 previous similar message [ 9038.815395] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9051.540908] Lustre: lustre-MDT0000: Client 07dd3627-4f21-47fc-8ab6-dbf936330522 (at 192.168.204.23@tcp) reconnected, waiting for 2 clients in recovery for 0:22 [ 9051.544411] Lustre: Skipped 1 previous similar message [ 9054.175587] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9054.391073] LustreError: 260904:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9054.398241] Lustre: 260904:0:(ldlm_lib.c:2931:target_recovery_thread()) too long recovery - read logs [ 9054.398366] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9054.402096] LustreError: dumping log to /tmp/lustre-log.1772478551.260904 [ 9054.403964] Lustre: Skipped 58 previous similar messages [ 9054.468227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13815 to 0x280000401:13833) [ 9054.468231] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13742 to 0x2c0000401:13761) [ 9055.742780] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9056.265040] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9059.554642] Lustre: DEBUG MARKER: == replay-single test complete, duration 8716 sec ======== 14:09:16 (1772478556) [ 9060.034828] Lustre: DEBUG MARKER: === replay-single: start cleanup 14:09:16 (1772478556) === [ 9062.474980] Lustre: DEBUG MARKER: === replay-single: finish cleanup 14:09:19 (1772478559) === [ 9063.080638] Lustre: Failing over lustre-MDT0000 [ 9064.415980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9064.417909] Lustre: Skipped 26 previous similar messages [ 9069.106173] Lustre: server umount lustre-MDT0000 complete [ 9083.970941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9084.018712] LustreError: MGC192.168.204.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9084.021976] LustreError: Skipped 8 previous similar messages [ 9084.143181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.145625] Lustre: Skipped 19 previous similar messages [ 9085.307248] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9089.526655] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13742 to 0x2c0000401:13793) [ 9089.526655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13815 to 0x280000401:13865) [ 9090.537924] Lustre: DEBUG MARKER: oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9090.993955] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9094.623608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9094.626093] LustreError: Skipped 15 previous similar messages [ 9095.311588] Lustre: server umount lustre-MDT0000 complete [ 9097.722633] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1772478594 with bad export cookie 10037793917224682384 [ 9097.725704] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9097.819804] Lustre: server umount lustre-MDT0001 complete [ 9110.596866] Lustre: server umount lustre-OST0000 complete [ 9122.323273] Lustre: server umount lustre-OST0001 complete [ 9127.399309] Lustre: DEBUG MARKER: oleg423-server.virtnet: executing unload_modules_local [ 9128.411955] Key type lgssc unregistered [ 9128.538441] LNet: 265489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9128.540470] LNetError: 265489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9128.551370] LNet: Removed LNI 192.168.204.123@tcp [ 9128.866102] Key type .llcrypt unregistered [ 9128.867072] Key type ._llcrypt unregistered