[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 522923342 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2464MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.002017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.003009] pid_max: default: 32768 minimum: 301 [ 0.004117] LSM: Security Framework initializing [ 0.005053] Yama: becoming mindful. [ 0.006034] SELinux: Initializing. [ 0.007080] *** VALIDATE selinux *** [ 0.017741] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023521] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025088] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027090] *** VALIDATE tmpfs *** [ 0.028492] *** VALIDATE proc *** [ 0.030104] *** VALIDATE cgroup *** [ 0.031005] *** VALIDATE cgroup2 *** [ 0.033140] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034122] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036030] Spectre V2 : User space: Vulnerable [ 0.037013] Speculative Store Bypass: Vulnerable [ 0.040423] debug: unmapping init [mem 0xffffffffbde59000-0xffffffffbde60fff] [ 0.042886] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043712] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044017] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046007] ... generic registers: 4 [ 0.047007] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051269] rcu: Hierarchical SRCU implementation. [ 0.053869] smp: Bringing up secondary CPUs ... [ 0.054783] x86: Booting SMP configuration: [ 0.055025] .... node #0, CPUs: #1 #2 #3 [ 0.058379] smp: Brought up 1 node, 4 CPUs [ 0.060007] smpboot: Max logical packages: 1 [ 0.061013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146042] node 0 deferred pages initialised in 83ms [ 0.152219] devtmpfs: initialized [ 0.155198] x86/mm: Memory block size: 128MB [ 0.158170] gcov: version magic: 0x41383552 [ 0.160308] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162293] pinctrl core: initialized pinctrl subsystem [ 0.163386] [ 0.163896] ************************************************************* [ 0.166011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168009] ** ** [ 0.171009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173009] ** ** [ 0.176010] ** This means that this kernel is built to expose internal ** [ 0.178009] ** IOMMU data structures, which may compromise security on ** [ 0.180010] ** your system. ** [ 0.183010] ** ** [ 0.185009] ** If you see this message and you are not debugging the ** [ 0.188010] ** kernel, report this immediately to your vendor! ** [ 0.190008] ** ** [ 0.192011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194054] ************************************************************* [ 0.197827] NET: Registered protocol family 16 [ 0.199486] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.205055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208590] cpuidle: using governor menu [ 0.210019] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212493] PCI: Using configuration type 1 for base access [ 0.215113] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228049] cryptd: max_cpu_qlen set to 1000 [ 0.231276] ACPI: Added _OSI(Module Device) [ 0.232000] ACPI: Added _OSI(Processor Device) [ 0.232000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241145] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249876] ACPI: Interpreter enabled [ 0.250136] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251011] ACPI: Using IOAPIC for interrupt routing [ 0.253242] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258534] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271043] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.273089] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.280080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.286428] acpiphp: Slot [2] registered [ 0.288291] acpiphp: Slot [5] registered [ 0.290133] acpiphp: Slot [6] registered [ 0.291213] acpiphp: Slot [7] registered [ 0.293095] acpiphp: Slot [8] registered [ 0.295132] acpiphp: Slot [9] registered [ 0.297105] acpiphp: Slot [10] registered [ 0.298129] acpiphp: Slot [3] registered [ 0.300077] acpiphp: Slot [4] registered [ 0.302096] acpiphp: Slot [11] registered [ 0.304082] acpiphp: Slot [12] registered [ 0.305151] acpiphp: Slot [13] registered [ 0.307077] acpiphp: Slot [14] registered [ 0.308000] acpiphp: Slot [15] registered [ 0.308000] acpiphp: Slot [16] registered [ 0.311135] acpiphp: Slot [17] registered [ 0.313074] acpiphp: Slot [18] registered [ 0.315077] acpiphp: Slot [19] registered [ 0.316207] acpiphp: Slot [20] registered [ 0.319101] acpiphp: Slot [21] registered [ 0.321081] acpiphp: Slot [22] registered [ 0.322079] acpiphp: Slot [23] registered [ 0.324084] acpiphp: Slot [24] registered [ 0.326093] acpiphp: Slot [25] registered [ 0.328158] acpiphp: Slot [26] registered [ 0.330092] acpiphp: Slot [27] registered [ 0.333106] acpiphp: Slot [28] registered [ 0.335078] acpiphp: Slot [29] registered [ 0.337092] acpiphp: Slot [30] registered [ 0.339109] acpiphp: Slot [31] registered [ 0.342083] PCI host bridge to bus 0000:00 [ 0.345046] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.348021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.350012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.353018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.356014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.359047] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.361148] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.364212] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368341] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.378743] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.385006] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.390021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.392012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.394012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.397478] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.399919] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.403040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.406776] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.413015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.428012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.434012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.440831] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.450020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.456023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.462000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.462000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.480000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.489015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.497013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.506015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.525486] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.538018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.545017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.562019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.567523] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.575021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.581022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.594000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.627739] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.636064] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.638000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.650020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.669441] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.671545] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.674510] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.677330] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.685260] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.691146] iommu: Default domain type: Passthrough [ 0.694394] SCSI subsystem initialized [ 0.696168] ACPI: bus type USB registered [ 0.701103] usbcore: registered new interface driver usbfs [ 0.704481] usbcore: registered new interface driver hub [ 0.707094] usbcore: registered new device driver usb [ 0.711207] pps_core: LinuxPPS API ver. 1 registered [ 0.714015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.720113] PTP clock support registered [ 0.762188] EDAC MC: Ver: 3.0.0 [ 0.765034] PCI: Using ACPI for IRQ routing [ 0.767488] NetLabel: Initializing [ 0.768000] NetLabel: domain hash size = 128 [ 0.768000] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.771105] NetLabel: unlabeled traffic allowed by default [ 0.774450] vgaarb: loaded [ 0.776135] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.778027] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.786253] clocksource: Switched to clocksource kvm-clock [ 0.918121] VFS: Disk quotas dquot_6.6.0 [ 0.919673] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.922690] *** VALIDATE ramfs *** [ 0.923507] *** VALIDATE hugetlbfs *** [ 0.924811] pnp: PnP ACPI init [ 0.927508] pnp: PnP ACPI: found 6 devices [ 0.942406] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.945869] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.948362] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.950707] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.953377] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.955973] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.959295] NET: Registered protocol family 2 [ 0.961834] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.966913] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.970767] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.976498] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.979945] TCP: Hash tables configured (established 65536 bind 65536) [ 0.983135] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.986404] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.989587] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.992839] NET: Registered protocol family 1 [ 0.995782] RPC: Registered named UNIX socket transport module. [ 1.002683] RPC: Registered udp transport module. [ 1.006457] RPC: Registered tcp transport module. [ 1.008451] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.010930] NET: Registered protocol family 44 [ 1.013117] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.015864] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.020432] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.024393] PCI: CLS 0 bytes, default 64 [ 1.027835] Unpacking initramfs... [ 2.902442] debug: unmapping init [mem 0xffff95b57cc54000-0xffff95b57ffbffff] [ 2.906624] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.909355] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.913895] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.608938] Initialise system trusted keyrings [ 3.614618] Key type blacklist registered [ 3.618344] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.631788] zbud: loaded [ 3.636837] *** VALIDATE nfs *** [ 3.639214] *** VALIDATE nfs4 *** [ 3.642232] pstore: using deflate compression [ 3.647870] Platform Keyring initialized [ 3.798235] NET: Registered protocol family 38 [ 3.803550] Key type asymmetric registered [ 3.806728] Asymmetric key parser 'x509' registered [ 3.810673] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.816751] io scheduler mq-deadline registered [ 3.819939] io scheduler kyber registered [ 3.823518] io scheduler bfq registered [ 3.827315] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.833795] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.840937] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.850441] ACPI: Power Button [PWRF] [ 3.858077] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.868754] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.897641] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.913191] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.953442] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.984390] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.017239] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.024234] Non-volatile memory driver v1.3 [ 4.027151] Linux agpgart interface v0.103 [ 4.060569] virtio_blk virtio1: [vda] 145792 512-byte logical blocks (74.6 MB/71.2 MiB) [ 4.065849] vda: detected capacity change from 0 to 74645504 [ 4.110356] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.124838] vdb: detected capacity change from 0 to 1073741824 [ 4.158854] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.163224] vdc: detected capacity change from 0 to 2621440000 [ 4.236745] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.240461] vdd: detected capacity change from 0 to 2621440000 [ 4.267495] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.270513] vde: detected capacity change from 0 to 4294967296 [ 4.303351] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.308148] vdf: detected capacity change from 0 to 4294967296 [ 4.511977] libphy: Fixed MDIO Bus: probed [ 4.518441] usbcore: registered new interface driver usbserial_generic [ 4.521843] usbserial: USB Serial support registered for generic [ 4.524670] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.530531] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.533312] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.538763] mousedev: PS/2 mouse device common for all mice [ 4.546611] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.560465] rtc_cmos 00:05: RTC can wake from S4 [ 4.571717] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.572157] rtc_cmos 00:05: registered as rtc0 [ 4.589419] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.596317] intel_pstate: CPU model not supported [ 4.600272] hid: raw HID events driver (C) Jiri Kosina [ 4.602366] usbcore: registered new interface driver usbhid [ 4.604889] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.609175] usbhid: USB HID core driver [ 4.611188] drop_monitor: Initializing network drop monitor service [ 4.613570] Initializing XFRM netlink socket [ 4.615533] NET: Registered protocol family 10 [ 4.618871] Segment Routing with IPv6 [ 4.620677] NET: Registered protocol family 17 [ 4.622967] mpls_gso: MPLS GSO support [ 4.629395] RAS: Correctable Errors collector initialized. [ 4.631156] AVX version of gcm_enc/dec engaged. [ 4.633101] AES CTR mode by8 optimization enabled [ 4.734819] sched_clock: Marking stable (4734745897, 0)->(5832453703, -1097707806) [ 4.744461] registered taskstats version 1 [ 4.746848] Loading compiled-in X.509 certificates [ 4.754731] zswap: loaded using pool lzo/zbud [ 4.815235] Key type big_key registered [ 4.829605] Key type encrypted registered [ 4.831406] ima: No TPM chip found, activating TPM-bypass! [ 4.833632] ima: Allocated hash algorithm: sha1 [ 4.835685] ima: No architecture policies found [ 4.837924] evm: Initialising EVM extended attributes: [ 4.840240] evm: security.selinux [ 4.841691] evm: security.ima [ 4.843472] evm: security.capability [ 4.845250] evm: HMAC attrs: 0x1 [ 4.848735] rtc_cmos 00:05: setting system clock to 2026-07-18 22:14:47 UTC (1784412887) [ 4.863812] debug: unmapping init [mem 0xffffffffbee03000-0xffffffffbeffffff] [ 4.871974] debug: unmapping init [mem 0xffffffffbdb82000-0xffffffffbde58fff] [ 4.882107] Write protecting the kernel read-only data: 28672k [ 4.885220] debug: unmapping init [mem 0xffffffffbc203000-0xffffffffbc3fffff] [ 4.888377] debug: unmapping init [mem 0xffffffffbcb14000-0xffffffffbcbfffff] [ 5.110405] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.121911] systemd[1]: Detected virtualization kvm. [ 5.124838] systemd[1]: Detected architecture x86-64. [ 5.128066] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.154569] systemd[1]: No hostname configured. [ 5.156339] systemd[1]: Set hostname to . [ 5.158437] random: systemd: uninitialized urandom read (16 bytes read) [ 5.161438] systemd[1]: Initializing machine ID from random generator. [ 5.259913] random: ln: uninitialized urandom read (6 bytes read) [ 5.413237] random: systemd: uninitialized urandom read (16 bytes read) [ 5.416073] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.430734] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.435877] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.787355] device-mapper: uevent: version 1.0.3 [ 6.789784] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.962781] virtio_net virtio0 ens2: renamed from eth0 [ 8.112131] random: fast init done [ 8.376502] scsi host0: ata_piix [ 8.436465] scsi host1: ata_piix [ 8.439748] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.448635] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.360288] random: crng init done [ 13.361883] random: 7 urandom warning(s) missed due to ratelimiting [ 14.232649] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.131436] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.968279] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.372175] SELinux: Disabled at runtime. [ 17.441685] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.450100] systemd[1]: Detected virtualization kvm. [ 17.451525] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.180658] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.187801] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.192440] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.197381] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.199766] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.208379] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.217809] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ 18.328225] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.445231] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.299675] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.314117] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.123480] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.217244] EDAC sbridge: Ver: 1.1.2 [ 23.994596] Key type dns_resolver registered [ 24.514855] NFS: Registering the id_resolver key type [ 24.518400] Key type id_resolver registered [ 24.521224] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg338-server login: [ 59.166644] libcfs: loading out-of-tree module taints kernel. [ 59.191475] Key type ._llcrypt registered [ 59.192853] Key type .llcrypt registered [ 59.251161] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_hostid [ 68.093143] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 68.743827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.753620] alg: No test for adler32 (adler32-zlib) [ 69.769488] Lustre: Lustre: Build Version: 2.17.54_173_ge83a086 [ 70.144180] LNet: Added LNI 192.168.203.138@tcp [8/256/0/180] [ 71.783242] Key type lgssc registered [ 72.414508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 79.458527] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.853186] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing load_modules_local [ 102.255332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 102.267365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 103.368714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 103.385977] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 103.429864] Lustre: lustre-MDT0000: new disk, initializing [ 103.464896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 103.473530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 105.142815] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.442205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 111.504092] Lustre: 6480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 111.524054] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 111.531396] Lustre: Skipped 1 previous similar message [ 111.581263] Lustre: lustre-MDT0001: new disk, initializing [ 111.615913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 111.630871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 111.641276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 113.359900] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 115.960680] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 119.892187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 120.008096] Lustre: lustre-OST0000: new disk, initializing [ 120.010288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 120.012719] Lustre: 8415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 120.032756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.071304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 121.074196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 121.096380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 122.144971] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.444637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 128.518252] Lustre: lustre-OST0001: new disk, initializing [ 128.520766] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 128.524762] Lustre: 9489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 128.568132] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 129.914054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 129.922903] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 129.953073] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 131.086867] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.650433] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 141.630837] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 144.025492] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing check_logdir /tmp/testlogs/ [ 145.922640] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing yml_node [ 147.819574] Lustre: DEBUG MARKER: Client: 2.17.54.173 [ 148.824475] Lustre: DEBUG MARKER: MDS: 2.17.54.173 [ 149.781259] Lustre: DEBUG MARKER: OSS: 2.17.54.173 [ 150.417658] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sat Jul 18 18:17:12 EDT 2026 [ 156.838179] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 157.549842] Lustre: DEBUG MARKER: === replay-single: start setup 18:17:19 (1784413039) === [ 159.734048] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing check_config_client /mnt/lustre [ 167.389460] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 168.673361] Lustre: 13291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 170.016158] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 171.945940] Lustre: DEBUG MARKER: === replay-single: finish setup 18:17:34 (1784413054) === [ 172.532673] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 18:17:34 (1784413054) [ 175.319840] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 176.061277] Lustre: Failing over lustre-MDT0000 [ 176.095716] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 176.096339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 176.101049] Lustre: Skipped 2 previous similar messages [ 176.103801] Lustre: Skipped 1 previous similar message [ 176.151928] Lustre: server umount lustre-MDT0000 complete [ 186.281418] LustreError: 6485:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 186.290612] LustreError: 6485:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 191.052390] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 191.055027] LDISKFS-fs (dm-0): recovery complete [ 191.065039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.091613] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 191.186047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.397192] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 192.945469] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.581777] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 196.605935] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 199.160798] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 199.873464] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 204.306502] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 18:18:06 (1784413086) [ 205.287440] Lustre: Failing over lustre-OST0000 [ 205.344241] Lustre: server umount lustre-OST0000 complete [ 206.774057] LustreError: 8595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 206.785452] LustreError: 8595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 206.816156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 206.819300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 206.825971] Lustre: Skipped 2 previous similar messages [ 211.882990] LustreError: 8595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 211.895265] LustreError: 8595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 216.999294] LustreError: 8595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 217.012968] LustreError: 8595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 219.472487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.558544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.100875] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 221.166670] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 221.166859] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 221.173514] Lustre: Skipped 3 previous similar messages [ 222.282788] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.258788] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 227.919401] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 232.271851] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 18:18:34 (1784413114) [ 235.159701] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 235.993662] Lustre: Failing over lustre-MDT0000 [ 236.103297] Lustre: server umount lustre-MDT0000 complete [ 237.536105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 237.541832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 237.550105] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 237.558897] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 240.095743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 240.103952] Lustre: Skipped 2 previous similar messages [ 250.336412] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 250.346358] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 250.963563] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 250.965351] LDISKFS-fs (dm-0): recovery complete [ 250.969681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 251.005659] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 251.098478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.695250] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.637080] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 253.640877] Lustre: lustre-MDT0000: Denying connection for new client f62dfa94-2fda-4004-a237-52835714d637 (at 192.168.203.38@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 256.483870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 256.487075] Lustre: Skipped 1 previous similar message [ 258.982869] Lustre: lustre-MDT0000: Denying connection for new client f62dfa94-2fda-4004-a237-52835714d637 (at 192.168.203.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 264.105279] Lustre: lustre-MDT0000: Denying connection for new client f62dfa94-2fda-4004-a237-52835714d637 (at 192.168.203.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 269.221271] Lustre: lustre-MDT0000: Denying connection for new client f62dfa94-2fda-4004-a237-52835714d637 (at 192.168.203.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 274.340750] Lustre: lustre-MDT0000: Denying connection for new client f62dfa94-2fda-4004-a237-52835714d637 (at 192.168.203.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 284.584455] Lustre: lustre-MDT0000: Denying connection for new client f62dfa94-2fda-4004-a237-52835714d637 (at 192.168.203.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 284.590891] Lustre: Skipped 1 previous similar message [ 305.060996] Lustre: lustre-MDT0000: Denying connection for new client f62dfa94-2fda-4004-a237-52835714d637 (at 192.168.203.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 305.069281] Lustre: Skipped 3 previous similar messages [ 323.502262] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 323.505650] Lustre: 18175:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d83aff8c-b6a6-41c6-b6e0-d90f03ba5a00@ [ 323.516732] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 323.526975] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 323.528932] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 323.537910] Lustre: Skipped 2 previous similar messages [ 323.547874] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 323.547882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 330.268996] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 18:20:12 (1784413212) [ 333.677205] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 334.573424] Lustre: Failing over lustre-MDT0000 [ 334.701363] Lustre: server umount lustre-MDT0000 complete [ 338.399834] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.400331] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.408577] Lustre: Skipped 3 previous similar messages [ 338.415699] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 350.667415] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 350.669453] LDISKFS-fs (dm-0): recovery complete [ 350.673237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.724950] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 350.834406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 352.877154] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 354.066756] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 354.072354] Lustre: lustre-MDT0000: Denying connection for new client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 354.082564] Lustre: Skipped 3 previous similar messages [ 356.325387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 420.773268] Lustre: lustre-MDT0000: Denying connection for new client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 420.784603] Lustre: Skipped 12 previous similar messages [ 424.500753] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 424.504227] Lustre: 19901:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f62dfa94-2fda-4004-a237-52835714d637@ [ 424.510726] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 424.517960] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 424.519021] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 424.523904] Lustre: Skipped 2 previous similar messages [ 424.537141] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 424.537261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 430.547991] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 18:21:52 (1784413312) [ 434.065890] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 434.860433] Lustre: Failing over lustre-MDT0000 [ 434.957672] Lustre: server umount lustre-MDT0000 complete [ 436.144755] LustreError: 6487:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 436.152151] LustreError: 6487:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 438.239810] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.246417] Lustre: Skipped 3 previous similar messages [ 450.075788] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 450.077200] LDISKFS-fs (dm-0): recovery complete [ 450.081628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 450.139934] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 450.236755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.495647] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 451.884847] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 455.654814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 455.667099] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 455.685191] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 455.685568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 457.988375] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 458.691893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 463.046497] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 18:22:25 (1784413345) [ 466.352269] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 467.208912] Lustre: Failing over lustre-MDT0000 [ 467.359923] Lustre: server umount lustre-MDT0000 complete [ 471.007478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.008220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 471.011325] Lustre: Skipped 3 previous similar messages [ 483.338921] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 483.340857] LDISKFS-fs (dm-0): recovery complete [ 483.348664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 483.396464] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.354855] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 487.339567] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 488.934587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 488.937735] Lustre: Skipped 3 previous similar messages [ 488.977731] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 489.001993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 489.002192] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 491.418079] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 492.173641] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 496.480429] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 18:22:58 (1784413378) [ 499.965388] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 500.852410] Lustre: Failing over lustre-MDT0000 [ 500.972695] Lustre: server umount lustre-MDT0000 complete [ 502.704279] LustreError: 6486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.713153] LustreError: 6486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [ 504.288574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.302458] Lustre: Skipped 2 previous similar messages [ 515.651677] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 515.653388] LDISKFS-fs (dm-0): recovery complete [ 515.657319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.701789] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 517.404767] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 518.053736] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 521.230284] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 521.253695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 521.257357] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:193) [ 523.770916] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 524.425505] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 528.764277] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 18:23:30 (1784413410) [ 531.997099] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 532.901070] Lustre: Failing over lustre-MDT0000 [ 533.093589] Lustre: server umount lustre-MDT0000 complete [ 536.543996] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 548.506216] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 548.511234] LDISKFS-fs (dm-0): recovery complete [ 548.515853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.574091] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.429635] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 553.954883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 553.958722] Lustre: Skipped 7 previous similar messages [ 554.013210] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 554.031688] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 554.031732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 556.320393] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 556.988343] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 560.816592] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 18:24:02 (1784413442) [ 563.901369] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 564.728410] Lustre: Failing over lustre-MDT0000 [ 564.836189] Lustre: server umount lustre-MDT0000 complete [ 569.311861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.318731] Lustre: Skipped 7 previous similar messages [ 569.321344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 579.659475] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 579.661422] LDISKFS-fs (dm-0): recovery complete [ 579.667455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.796913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.800474] Lustre: Skipped 3 previous similar messages [ 581.371378] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 584.625368] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 584.628423] Lustre: Skipped 1 previous similar message [ 585.220536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 585.220853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 587.399072] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 588.028170] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 591.892173] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 18:24:33 (1784413473) [ 592.314554] Lustre: *** cfs_fail_loc=13b, val=315*** [ 592.316734] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 592.318310] LustreError: 6488:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b5ff884700 x1871092587287552/t38654705666(0) o35->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:430/0 lens 392/456 e 0 to 0 dl 1784413485 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 596.524299] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 597.309366] Lustre: Failing over lustre-MDT0000 [ 597.479894] Lustre: server umount lustre-MDT0000 complete [ 600.543609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 612.550700] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 612.552453] LDISKFS-fs (dm-0): recovery complete [ 612.558172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.628808] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.635139] LustreError: Skipped 1 previous similar message [ 612.736421] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 614.589824] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 617.984168] Lustre: 6488:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b4c4190700 x1871092587287552/t38654705666(0) o35->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:456/0 lens 392/456 e 0 to 0 dl 1784413511 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 618.000892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 618.001092] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 620.353262] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 620.993362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 624.689832] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 18:25:06 (1784413506) [ 627.444497] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 628.197805] Lustre: Failing over lustre-MDT0000 [ 628.346234] Lustre: server umount lustre-MDT0000 complete [ 630.704339] LustreError: 20278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 630.710833] LustreError: 20278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 59 previous similar messages [ 633.311951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 643.167586] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 643.169587] LDISKFS-fs (dm-0): recovery complete [ 643.174703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.343582] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 644.971575] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 648.696528] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 648.701190] Lustre: Skipped 2 previous similar messages [ 648.717232] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 648.717455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 650.824756] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 651.457967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 655.266219] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 18:25:37 (1784413537) [ 658.183761] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 658.548214] Lustre: *** cfs_fail_loc=114, val=0*** [ 659.575568] Lustre: Failing over lustre-MDT0000 [ 659.737887] Lustre: server umount lustre-MDT0000 complete [ 674.869088] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 674.870497] LDISKFS-fs (dm-0): recovery complete [ 674.877516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.075228] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 676.741837] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 676.774121] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 676.779330] Lustre: Skipped 2 previous similar messages [ 680.455694] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 680.455803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 682.628403] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 683.351370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 687.461768] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 18:26:09 (1784413569) [ 690.584993] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 691.013490] Lustre: *** cfs_fail_loc=128, val=0*** [ 692.181323] Lustre: Failing over lustre-MDT0000 [ 692.320028] Lustre: server umount lustre-MDT0000 complete [ 695.778199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 695.783195] LustreError: Skipped 1 previous similar message [ 706.965457] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 706.967312] LDISKFS-fs (dm-0): recovery complete [ 706.973730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.027441] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 707.032091] LustreError: Skipped 2 previous similar messages [ 707.126314] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 708.682194] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 712.163829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 712.166102] Lustre: Skipped 19 previous similar messages [ 712.196569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 712.196627] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 714.227129] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 714.802034] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 718.606349] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 18:26:40 (1784413600) [ 721.455092] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 722.266972] Lustre: Failing over lustre-MDT0000 [ 722.399396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.406216] Lustre: Skipped 17 previous similar messages [ 722.425310] Lustre: server umount lustre-MDT0000 complete [ 737.292683] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 737.294381] LDISKFS-fs (dm-0): recovery complete [ 737.299669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.497106] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 739.112389] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 742.972261] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 742.972695] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 745.094270] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 745.721915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 749.708662] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 18:27:11 (1784413631) [ 752.841935] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 753.691166] Lustre: Failing over lustre-MDT0000 [ 753.871436] Lustre: server umount lustre-MDT0000 complete [ 768.566137] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 768.568171] LDISKFS-fs (dm-0): recovery complete [ 768.573038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.744874] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 770.270199] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 774.167451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 774.167456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 776.184103] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 776.745934] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 780.475531] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 18:27:42 (1784413662) [ 783.261983] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 786.030352] Lustre: Failing over lustre-MDT0000 [ 786.262451] Lustre: server umount lustre-MDT0000 complete [ 789.472335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 789.476721] LustreError: Skipped 2 previous similar messages [ 800.818827] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 800.820985] LDISKFS-fs (dm-0): recovery complete [ 800.824801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 800.966060] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 802.449451] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 807.317883] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 807.321897] Lustre: Skipped 4 previous similar messages [ 807.336760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 807.336786] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 809.388757] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 810.010728] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 820.628616] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 18:28:22 (1784413702) [ 823.315132] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 823.962945] Lustre: Failing over lustre-MDT0000 [ 824.129483] Lustre: server umount lustre-MDT0000 complete [ 838.294423] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 838.296442] LDISKFS-fs (dm-0): recovery complete [ 838.299927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.336342] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 838.340375] LustreError: Skipped 3 previous similar messages [ 838.413557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 838.416641] Lustre: Skipped 7 previous similar messages [ 838.432118] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 839.790514] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 840.612699] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 840.616076] Lustre: Skipped 4 previous similar messages [ 843.769469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 843.769470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 845.520565] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 846.020575] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 851.366583] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 18:28:53 (1784413733) [ 853.860391] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 854.500938] Lustre: Failing over lustre-MDT0000 [ 854.628458] Lustre: server umount lustre-MDT0000 complete [ 868.837615] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 868.839896] LDISKFS-fs (dm-0): recovery complete [ 868.843808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.418080] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 874.494443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 874.494482] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 876.374192] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 876.896395] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 880.213617] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 18:29:22 (1784413762) [ 882.754505] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 883.407128] Lustre: Failing over lustre-MDT0000 [ 883.558629] Lustre: server umount lustre-MDT0000 complete [ 889.823534] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.829680] LustreError: 6491:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 117 previous similar messages [ 897.607811] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 897.609181] LDISKFS-fs (dm-0): recovery complete [ 897.613299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.107183] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 903.161896] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 903.161925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 904.938542] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 905.471391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 908.882387] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 18:29:51 (1784413791) [ 911.345158] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 912.033163] Lustre: Failing over lustre-MDT0000 [ 912.239250] Lustre: server umount lustre-MDT0000 complete [ 926.306259] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 926.307892] LDISKFS-fs (dm-0): recovery complete [ 926.310885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 926.442129] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 926.445204] Lustre: Skipped 2 previous similar messages [ 927.766930] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 931.847885] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 931.847903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 933.657679] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 934.181981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 937.609732] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 18:30:19 (1784413819) [ 940.071035] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 940.734910] Lustre: Failing over lustre-MDT0000 [ 940.959605] Lustre: server umount lustre-MDT0000 complete [ 942.047832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 942.051144] LustreError: Skipped 3 previous similar messages [ 955.171157] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 955.173267] LDISKFS-fs (dm-0): recovery complete [ 955.176892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.629377] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 960.508253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 960.508254] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 962.294690] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 962.874105] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 966.487294] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 18:30:48 (1784413848) [ 969.137788] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 969.907841] Lustre: Failing over lustre-MDT0000 [ 970.112091] Lustre: server umount lustre-MDT0000 complete [ 984.597854] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 984.599835] LDISKFS-fs (dm-0): recovery complete [ 984.604703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.059160] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 990.177250] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 990.180647] Lustre: Skipped 35 previous similar messages [ 990.204074] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 990.204097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 992.034471] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 992.547620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 995.977957] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 18:31:18 (1784413878) [ 998.482972] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 999.095931] Lustre: Failing over lustre-MDT0000 [ 999.288943] Lustre: server umount lustre-MDT0000 complete [ 1000.415694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.418747] Lustre: Skipped 38 previous similar messages [ 1013.229330] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1013.230438] LDISKFS-fs (dm-0): recovery complete [ 1013.233505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1014.566255] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1018.378679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1018.378689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1020.173053] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1020.700757] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1024.122164] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 18:31:46 (1784413906) [ 1026.598797] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1027.233531] Lustre: Failing over lustre-MDT0000 [ 1027.392353] Lustre: server umount lustre-MDT0000 complete [ 1041.471445] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1041.474120] LDISKFS-fs (dm-0): recovery complete [ 1041.478091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.929861] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1047.031752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1047.031934] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1048.843668] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1049.337619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1052.854539] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 18:32:14 (1784413934) [ 1055.480569] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1056.119900] Lustre: Failing over lustre-MDT0000 [ 1056.251718] Lustre: server umount lustre-MDT0000 complete [ 1070.364636] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1070.366373] LDISKFS-fs (dm-0): recovery complete [ 1070.371341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.479651] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1070.482569] Lustre: Skipped 4 previous similar messages [ 1071.819956] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1075.687754] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1075.690839] Lustre: Skipped 8 previous similar messages [ 1075.706708] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1075.706725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1077.468180] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1077.976836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1081.321626] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 18:32:43 (1784413963) [ 1083.789240] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1084.440129] Lustre: Failing over lustre-MDT0000 [ 1084.585293] Lustre: server umount lustre-MDT0000 complete [ 1098.454923] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1098.456136] LDISKFS-fs (dm-0): recovery complete [ 1098.459594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.492456] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1098.496672] LustreError: Skipped 8 previous similar messages [ 1099.810296] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1101.732414] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1101.735710] Lustre: Skipped 8 previous similar messages [ 1103.861792] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:897) [ 1103.861792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 1105.595821] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1106.114582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1109.435013] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 18:33:11 (1784413991) [ 1112.026170] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1112.693020] Lustre: Failing over lustre-MDT0000 [ 1112.847317] Lustre: server umount lustre-MDT0000 complete [ 1126.928068] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1126.929215] LDISKFS-fs (dm-0): recovery complete [ 1126.931524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.330620] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1132.537508] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1132.537519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1134.250334] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1134.746964] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1137.949361] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 18:33:40 (1784414020) [ 1140.317881] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1140.951860] Lustre: Failing over lustre-MDT0000 [ 1141.083927] Lustre: server umount lustre-MDT0000 complete [ 1154.882893] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1154.883878] LDISKFS-fs (dm-0): recovery complete [ 1154.886636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.194077] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1160.185186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 1160.185189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 1161.843655] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1162.312582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1165.493305] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 18:34:07 (1784414047) [ 1167.818537] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1168.451883] Lustre: Failing over lustre-MDT0000 [ 1168.592576] Lustre: server umount lustre-MDT0000 complete [ 1182.177463] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1182.179087] LDISKFS-fs (dm-0): recovery complete [ 1182.181969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1183.418268] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1187.315819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 1187.315840] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:993) [ 1188.906371] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1189.365034] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1192.480838] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 18:34:34 (1784414074) [ 1194.666020] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1195.268922] Lustre: Failing over lustre-MDT0000 [ 1195.420329] Lustre: server umount lustre-MDT0000 complete [ 1209.266111] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1209.267882] LDISKFS-fs (dm-0): recovery complete [ 1209.270848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.650373] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1214.462318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1214.462363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1216.226847] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1216.770475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1219.966402] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 18:35:02 (1784414102) [ 1222.223437] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1222.817143] Lustre: Failing over lustre-MDT0000 [ 1222.942325] Lustre: server umount lustre-MDT0000 complete [ 1224.671485] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1224.673458] LustreError: Skipped 9 previous similar messages [ 1236.485801] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1236.487407] LDISKFS-fs (dm-0): recovery complete [ 1236.490700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1237.779521] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1242.106861] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1242.106869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1243.693538] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1244.153787] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1247.371477] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:35:29 (1784414129) [ 1249.753687] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1250.383724] Lustre: Failing over lustre-MDT0000 [ 1250.521203] Lustre: server umount lustre-MDT0000 complete [ 1264.213800] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1264.215502] LDISKFS-fs (dm-0): recovery complete [ 1264.218424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.504843] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1269.747813] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 1269.747839] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 1271.305203] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1271.812331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1275.199598] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 18:35:57 (1784414157) [ 1276.808126] Lustre: 75067:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c2195709-2ef6-4414-8a03-c1f4c705ea9b at adminstrative request [ 1280.204676] Lustre: Failing over lustre-MDT0000 [ 1280.386283] Lustre: server umount lustre-MDT0000 complete [ 1292.843844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.073812] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1298.423380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1298.423388] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 1300.203101] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1300.707622] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1302.934132] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1305.451802] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1307.246316] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 18:36:29 (1784414189) [ 1307.546167] Lustre: 77090:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c2195709-2ef6-4414-8a03-c1f4c705ea9b at adminstrative request [ 1311.871195] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 18:36:34 (1784414194) [ 1314.100067] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1314.722937] Lustre: Failing over lustre-MDT0000 [ 1314.957685] Lustre: server umount lustre-MDT0000 complete [ 1328.539696] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1328.540706] LDISKFS-fs (dm-0): recovery complete [ 1328.542770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.641195] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1328.643301] Lustre: Skipped 8 previous similar messages [ 1329.745873] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1333.765978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 1333.765985] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 1335.252868] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1335.690632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1339.035977] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:37:01 (1784414221) [ 1341.346352] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1341.910630] Lustre: Failing over lustre-MDT0000 [ 1342.108952] Lustre: server umount lustre-MDT0000 complete [ 1355.726120] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1355.727639] LDISKFS-fs (dm-0): recovery complete [ 1355.730884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1355.823620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1355.825598] Lustre: Skipped 17 previous similar messages [ 1357.016244] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1360.884727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1360.884727] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1362.419518] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1362.859602] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1366.049801] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 18:37:28 (1784414248) [ 1368.290789] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1368.862947] Lustre: Failing over lustre-MDT0000 [ 1369.026732] Lustre: server umount lustre-MDT0000 complete [ 1382.621937] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1382.623390] LDISKFS-fs (dm-0): recovery complete [ 1382.625823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.851756] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1388.020861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1388.020872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1389.541384] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1390.007604] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1393.065870] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 18:37:55 (1784414275) [ 1395.284190] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1395.822537] Lustre: Failing over lustre-MDT0000 [ 1395.997275] Lustre: server umount lustre-MDT0000 complete [ 1403.359493] LustreError: 7521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.364484] LustreError: 7521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 256 previous similar messages [ 1409.696415] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1409.697799] LDISKFS-fs (dm-0): recovery complete [ 1409.700632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.073789] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1415.159335] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1415.160017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1416.843673] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1417.298540] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1420.595927] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:38:22 (1784414302) [ 1422.977241] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1423.577040] Lustre: Failing over lustre-MDT0000 [ 1423.743572] Lustre: server umount lustre-MDT0000 complete [ 1437.236062] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1437.237593] LDISKFS-fs (dm-0): recovery complete [ 1437.239977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.516265] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1442.801335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 1442.801339] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 1444.443939] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1444.943932] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1448.163769] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 18:38:50 (1784414330) [ 1450.535230] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1451.208076] Lustre: Failing over lustre-MDT0000 [ 1451.387486] Lustre: server umount lustre-MDT0000 complete [ 1465.146061] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1465.147242] LDISKFS-fs (dm-0): recovery complete [ 1465.149837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1466.408160] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1470.459610] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1470.459617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1472.051684] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1472.554903] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1475.793558] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:39:17 (1784414357) [ 1478.026803] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1478.600667] Lustre: Failing over lustre-MDT0000 [ 1478.727158] Lustre: server umount lustre-MDT0000 complete [ 1492.409254] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1492.411329] LDISKFS-fs (dm-0): recovery complete [ 1492.414536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.740085] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1497.602476] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1497.602476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1499.154111] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1499.586076] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1502.713744] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 18:39:44 (1784414384) [ 1504.944469] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1505.504956] Lustre: Failing over lustre-MDT0000 [ 1505.665195] Lustre: server umount lustre-MDT0000 complete [ 1519.601273] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1519.602318] LDISKFS-fs (dm-0): recovery complete [ 1519.605577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.874016] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1524.705018] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1524.707265] Lustre: Skipped 75 previous similar messages [ 1524.728293] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1524.729329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1526.286808] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1526.741849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1529.875632] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:40:12 (1784414412) [ 1532.113882] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1532.697714] Lustre: Failing over lustre-MDT0000 [ 1532.830269] Lustre: server umount lustre-MDT0000 complete [ 1534.943661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.948173] Lustre: Skipped 75 previous similar messages [ 1546.476180] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1546.477166] LDISKFS-fs (dm-0): recovery complete [ 1546.480187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.728750] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1551.863301] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1551.863339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1553.572180] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1554.090068] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1557.443601] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:40:39 (1784414439) [ 1559.983641] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1560.614288] Lustre: Failing over lustre-MDT0000 [ 1560.820975] Lustre: server umount lustre-MDT0000 complete [ 1574.843022] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1574.845301] LDISKFS-fs (dm-0): recovery complete [ 1574.848988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1576.248950] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1580.020494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1580.020494] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1581.732820] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1582.223507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1585.684500] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 18:41:07 (1784414467) [ 1588.241404] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1588.851318] Lustre: Failing over lustre-MDT0000 [ 1589.037160] Lustre: server umount lustre-MDT0000 complete [ 1602.878074] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1602.880308] LDISKFS-fs (dm-0): recovery complete [ 1602.883935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.222588] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1608.167254] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1608.169884] Lustre: Skipped 18 previous similar messages [ 1608.183900] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 1608.183912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 1609.866369] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1610.339992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1613.661407] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 18:41:35 (1784414495) [ 1614.019345] Lustre: 98593:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c2195709-2ef6-4414-8a03-c1f4c705ea9b at adminstrative request [ 1618.549284] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 18:41:40 (1784414500) [ 1620.695561] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1621.272983] Lustre: Failing over lustre-MDT0000 [ 1621.442476] Lustre: server umount lustre-MDT0000 complete [ 1625.236229] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1625.238497] LDISKFS-fs (dm-0): recovery complete [ 1625.243394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.274517] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1625.278633] LustreError: Skipped 18 previous similar messages [ 1625.350949] Lustre: lustre-MDT0000: Aborting client recovery [ 1625.352182] LustreError: 99873:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1625.354340] Lustre: 99906:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1625.357427] Lustre: 99906:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c2195709-2ef6-4414-8a03-c1f4c705ea9b@ [ 1625.361875] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1625.364457] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1625.367869] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1625.387489] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 1625.387539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 1626.749689] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1630.690120] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1638.670789] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 18:42:00 (1784414520) [ 1641.155240] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1641.770585] Lustre: Failing over lustre-MDT0000 [ 1641.933202] Lustre: server umount lustre-MDT0000 complete [ 1646.102103] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1646.104502] LDISKFS-fs (dm-0): recovery complete [ 1646.108961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.225112] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1646.263868] Lustre: lustre-MDT0000: Aborting client recovery [ 1646.266362] LustreError: 101734:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1646.271823] Lustre: 101767:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1646.274868] Lustre: 101767:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 1646.277425] Lustre: 101767:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1646.281333] Lustre: 101767:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1646.283925] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1646.287313] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 1646.293549] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 1646.316532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1537) [ 1646.316561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1537) [ 1647.646131] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1651.681181] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1656.901790] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1656.903594] Lustre: Skipped 1 previous similar message [ 1659.108548] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 18:42:21 (1784414541) [ 1661.418949] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1661.970966] Lustre: Failing over lustre-MDT0000 [ 1662.049340] Lustre: server umount lustre-MDT0000 complete [ 1665.840131] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1665.841329] LDISKFS-fs (dm-0): recovery complete [ 1665.843651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.950890] Lustre: lustre-MDT0000: Aborting client recovery [ 1665.952687] LustreError: 103598:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1665.955387] Lustre: 103632:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1665.959025] Lustre: 103632:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 1665.961817] Lustre: 103632:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1665.965334] Lustre: 103632:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1665.968026] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1665.970652] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 1665.975617] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 1665.992645] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 1665.992781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 1667.229689] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1671.136337] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1679.167663] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 18:42:41 (1784414561) [ 1679.468638] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1679.470199] LustreError: 20278:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b5ff5a0380 x1871092588205056/t201863462916(0) o36->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:8/0 lens 512/456 e 0 to 0 dl 1784414573 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1682.042809] Lustre: Failing over lustre-MDT0000 [ 1682.129386] Lustre: server umount lustre-MDT0000 complete [ 1685.013092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1685.148053] Lustre: lustre-MDT0000: Aborting client recovery [ 1685.150091] LustreError: 104920:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1685.153677] Lustre: 104954:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1685.156576] Lustre: 104954:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 1685.158698] Lustre: 104954:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1685.161390] Lustre: 104954:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1685.163606] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1685.166076] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 1685.171831] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 1685.191064] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 1685.191108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1601) [ 1686.522134] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1690.592899] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1697.781047] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1698.313020] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 18:43:00 (1784414580) [ 1700.848084] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1701.780738] Lustre: Failing over lustre-MDT0000 [ 1701.967149] Lustre: server umount lustre-MDT0000 complete [ 1705.977877] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1705.979044] LDISKFS-fs (dm-0): recovery complete [ 1705.981561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.095421] Lustre: lustre-MDT0000: Aborting client recovery [ 1706.098208] LustreError: 106881:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1706.100931] Lustre: 106914:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1706.104843] Lustre: 106914:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 1706.107907] Lustre: 106914:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1706.111864] Lustre: 106914:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1706.114680] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1706.117562] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 1706.123640] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 1706.143504] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 1706.143505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1633) [ 1707.409228] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1711.586281] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1720.799561] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:43:22 (1784414602) [ 1728.286051] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1728.856516] Lustre: Failing over lustre-MDT0000 [ 1729.011162] Lustre: server umount lustre-MDT0000 complete [ 1742.907764] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1742.909271] LDISKFS-fs (dm-0): recovery complete [ 1742.912114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.505198] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1746.852418] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1746.857782] Lustre: Skipped 18 previous similar messages [ 1748.492639] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 1748.492670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 1750.358246] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1750.885231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1758.461665] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:44:00 (1784414640) [ 1764.203835] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1766.637747] Lustre: Failing over lustre-MDT0000 [ 1766.833913] Lustre: server umount lustre-MDT0000 complete [ 1768.927793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1768.930350] LustreError: Skipped 17 previous similar messages [ 1780.721859] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1780.724007] LDISKFS-fs (dm-0): recovery complete [ 1780.728409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.179687] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1786.913954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 1786.914021] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 1788.680310] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1789.204829] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1796.914307] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 18:44:39 (1784414679) [ 1797.611147] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1797.928146] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1797.930875] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1799.978294] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 18:44:42 (1784414682) [ 1805.898724] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1808.662921] Lustre: Failing over lustre-OST0000 [ 1808.711595] Lustre: server umount lustre-OST0000 complete [ 1822.777086] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 1822.778941] LDISKFS-fs (dm-2): recovery complete [ 1822.782376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1824.715076] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1870.251946] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 18:45:52 (1784414752) [ 1872.996219] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1873.987120] Lustre: Failing over lustre-MDT0000 [ 1874.285015] Lustre: server umount lustre-MDT0000 complete [ 1888.222332] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1888.223554] LDISKFS-fs (dm-0): recovery complete [ 1888.226534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.364369] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1888.366631] Lustre: Skipped 28 previous similar messages [ 1889.562925] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1893.372118] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 1893.372146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 1895.094500] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1895.589798] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1909.113889] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 18:46:31 (1784414791) [ 1909.727095] Lustre: 116227:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784414776/real 1784414776] req@ffff95b4c5683100 x1871092597347840/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1784414792 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 1909.735425] LustreError: 116227:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 1909.736709] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1910.751530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 1910.823746] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1915.871091] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1915.874702] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnecting [ 1916.469991] LustreError: 12413:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1921.503109] LustreError: 12413:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1921.506482] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnecting [ 1922.122753] LustreError: 7435:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1927.135092] LustreError: 7435:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1927.137927] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnecting [ 1927.755460] LustreError: 20278:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1932.767137] LustreError: 20278:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1933.368135] LustreError: 12413:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1938.399109] LustreError: 12413:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1938.402658] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnecting [ 1938.406316] Lustre: Skipped 1 previous similar message [ 1944.659912] LustreError: 7435:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1944.663280] LustreError: 7435:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1950.175128] LustreError: 7435:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1950.177814] LustreError: 7435:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1956.319095] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnecting [ 1956.321601] Lustre: Skipped 2 previous similar messages [ 1962.533686] LustreError: 6485:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1962.535972] LustreError: 6485:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1967.583113] LustreError: 6485:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1967.586252] LustreError: 6485:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1970.526741] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 18:47:32 (1784414852) [ 1971.191189] LustreError: 117305:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1981.348990] Lustre: lustre-MDT0000: Export ffff95b5e7abb000 already connecting from 192.168.203.38@tcp [ 1986.468465] Lustre: lustre-MDT0000: Export ffff95b5e7abb000 already connecting from 192.168.203.38@tcp [ 1991.588509] Lustre: lustre-MDT0000: Export ffff95b5e7abb000 already connecting from 192.168.203.38@tcp [ 1996.708379] Lustre: lustre-MDT0000: Export ffff95b5e7abb000 already connecting from 192.168.203.38@tcp [ 1996.710594] Lustre: Skipped 1 previous similar message [ 2001.828390] Lustre: lustre-MDT0000: Export ffff95b5e7abb000 already connecting from 192.168.203.38@tcp [ 2011.239071] LustreError: 117305:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2011.242300] Lustre: 117305:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff95b5d18f4e00 x1871092590833408/t0(0) o38->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:0/0 lens 520/416 e 0 to 0 dl 1784414873 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2012.068371] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnecting [ 2012.071535] Lustre: Skipped 3 previous similar messages [ 2012.072697] LustreError: 6487:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2036.644420] Lustre: lustre-MDT0000: Export ffff95b5e7abb000 already connecting from 192.168.203.38@tcp [ 2036.647053] Lustre: Skipped 1 previous similar message [ 2052.119096] LustreError: 6487:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2052.122061] Lustre: 6487:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff95b5f4ced500 x1871092590838144/t0(0) o38->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:0/0 lens 520/416 e 0 to 0 dl 1784414914 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2055.109903] LustreError: 7435:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2079.652412] Lustre: lustre-MDT0000: Export ffff95b5e7abb000 already connecting from 192.168.203.38@tcp [ 2079.654829] Lustre: Skipped 3 previous similar messages [ 2095.151142] LustreError: 7435:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2095.153347] Lustre: 7435:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff95b5ffbeb480 x1871092590841728/t0(0) o38->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:0/0 lens 520/416 e 0 to 0 dl 1784414957 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2097.141109] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnecting [ 2097.143399] Lustre: Skipped 1 previous similar message [ 2097.144499] LustreError: 6487:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2121.636268] Lustre: lustre-MDT0000: Export ffff95b5e7abb000 already connecting from 192.168.203.38@tcp [ 2121.638671] Lustre: Skipped 3 previous similar messages [ 2137.183072] LustreError: 6487:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2137.187243] Lustre: 6487:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff95b5f5384e00 x1871092590845184/t0(0) o38->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:0/0 lens 520/416 e 0 to 0 dl 1784414999 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2139.133301] LustreError: 20278:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2179.175070] LustreError: 20278:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2179.178951] Lustre: 20278:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff95b5e7905500 x1871092590848640/t0(0) o38->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:0/0 lens 520/416 e 0 to 0 dl 1784415041 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2182.496923] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 18:51:04 (1784415064) [ 2186.172739] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2187.299503] Lustre: Failing over lustre-MDT0000 [ 2187.393951] Lustre: server umount lustre-MDT0000 complete [ 2189.221532] LustreError: 6486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.226489] LustreError: 6486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 178 previous similar messages [ 2190.304433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2190.307869] Lustre: Skipped 44 previous similar messages [ 2191.323606] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2191.325639] LDISKFS-fs (dm-0): recovery complete [ 2191.329186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2191.421976] Lustre: *** cfs_fail_loc=712, val=0*** [ 2191.424377] LustreError: 43756:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff95b5f53a5c00 x1871092597494016/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2191.435818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.437741] Lustre: Skipped 18 previous similar messages [ 2191.476378] Lustre: lustre-MDT0000: Aborting client recovery [ 2191.477842] LustreError: 121071:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2191.480036] Lustre: 121103:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2191.482340] Lustre: 121103:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2191.484402] Lustre: 121103:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2191.487270] Lustre: 121103:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2191.489336] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2191.491880] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2191.496962] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2191.516698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2191.516926] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2192.833118] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2196.449252] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2196.451230] Lustre: Skipped 51 previous similar messages [ 2196.453936] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2202.581551] Lustre: Failing over lustre-MDT0000 [ 2202.794374] Lustre: server umount lustre-MDT0000 complete [ 2215.430645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.713261] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2221.032640] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2221.036170] Lustre: Skipped 4 previous similar messages [ 2221.052854] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2221.054822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2222.715030] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2223.209821] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2226.450437] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 18:51:48 (1784415108) [ 2226.485939] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnecting [ 2226.489898] Lustre: Skipped 2 previous similar messages [ 2229.578065] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 18:51:51 (1784415111) [ 2229.887657] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2229.888924] LustreError: 100410:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b5f5384e00 x1871092590904704/t0(0) o700->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:558/0 lens 264/248 e 0 to 0 dl 1784415123 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2247.038883] Lustre: Failing over lustre-MDT0000 [ 2247.143201] Lustre: server umount lustre-MDT0000 complete [ 2259.663454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.698716] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2259.702869] LustreError: Skipped 9 previous similar messages [ 2261.067601] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2265.073764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2265.073781] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2266.697661] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2267.166539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2271.020678] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:52:33 (1784415153) [ 2271.802810] Lustre: Failing over lustre-OST0000 [ 2271.851795] Lustre: server umount lustre-OST0000 complete [ 2284.504661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.473581] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2290.163146] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2290.685939] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2356.311000] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:53:58 (1784415238) [ 2358.828175] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2359.471616] Lustre: Failing over lustre-MDT0000 [ 2359.633820] Lustre: server umount lustre-MDT0000 complete [ 2373.441571] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2373.442755] LDISKFS-fs (dm-0): recovery complete [ 2373.445287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2374.777849] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2378.148254] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2378.151064] Lustre: Skipped 6 previous similar messages [ 2378.803854] Lustre: *** cfs_fail_loc=216, val=0*** [ 2378.803877] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2378.805850] LustreError: 127219:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2379.871417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2441.004281] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 18:55:23 (1784415323) [ 2448.684354] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 18:55:30 (1784415330) [ 2449.535040] Lustre: Failing over lustre-MDT0000 [ 2449.686482] Lustre: server umount lustre-MDT0000 complete [ 2450.399982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2450.402049] LustreError: Skipped 6 previous similar messages [ 2462.327453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.723841] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2467.815158] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2467.816499] LustreError: 128799:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b4c236c380 x1871092591023232/t0(0) o101->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:82/0 lens 328/344 e 0 to 0 dl 1784415402 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2525.092241] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnected, waiting for 2 clients in recovery for 1:05 [ 2525.099482] Lustre: 128799:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2525.123245] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 2525.123291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 2527.033033] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2527.542013] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2531.097607] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 18:56:53 (1784415413) [ 2532.466266] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2535.026546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2535.610201] Lustre: Failing over lustre-MDT0000 [ 2535.769288] Lustre: server umount lustre-MDT0000 complete [ 2549.563057] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2549.564144] LDISKFS-fs (dm-0): recovery complete [ 2549.567135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.707185] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2549.709107] Lustre: Skipped 8 previous similar messages [ 2550.919627] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2554.865744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 2554.865776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3137) [ 2556.605380] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2557.102269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2560.400699] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 18:57:22 (1784415442) [ 2560.745424] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2564.265715] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2564.863145] Lustre: Failing over lustre-MDT0000 [ 2564.945908] Lustre: server umount lustre-MDT0000 complete [ 2578.809356] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2578.810456] LDISKFS-fs (dm-0): recovery complete [ 2578.812930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.182936] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2584.052580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 2584.052585] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3137) [ 2585.683648] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2586.164670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2589.455874] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 18:57:51 (1784415471) [ 2589.783247] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2593.334483] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2593.926711] Lustre: Failing over lustre-MDT0000 [ 2594.008366] Lustre: server umount lustre-MDT0000 complete [ 2607.836415] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2607.837657] LDISKFS-fs (dm-0): recovery complete [ 2607.840230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.198917] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2613.234871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 2613.234915] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3169) [ 2617.502687] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 18:58:19 (1784415499) [ 2618.856086] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2618.858504] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2618.861170] LustreError: 38536:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b5f4cece00 x1871092591074560/t257698037777(0) o35->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:192/0 lens 392/456 e 0 to 0 dl 1784415512 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2619.729976] Lustre: Failing over lustre-MDT0000 [ 2619.938180] Lustre: server umount lustre-MDT0000 complete [ 2632.642640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.082946] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2637.807379] Lustre: 6488:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5f4ebf100 x1871092591074560/t257698037777(0) o35->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:211/0 lens 392/456 e 0 to 0 dl 1784415531 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2637.812808] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 2637.814091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3233) [ 2639.500633] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2639.959837] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2643.158087] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 18:58:45 (1784415525) [ 2643.499748] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2643.501357] LustreError: 6486:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b5d18f5180 x1871092591087104/t261993005072(0) o36->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:261/0 lens 504/448 e 0 to 0 dl 1784415581 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2647.122770] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2647.712144] Lustre: Failing over lustre-MDT0000 [ 2647.795682] Lustre: server umount lustre-MDT0000 complete [ 2661.562776] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2661.564438] LDISKFS-fs (dm-0): recovery complete [ 2661.567293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.943971] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2666.988994] Lustre: 6485:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5f53d8700 x1871092591087104/t261993005072(0) o36->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:284/0 lens 504/2880 e 0 to 0 dl 1784415604 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2666.994184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3265) [ 2666.994287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 2668.650752] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2669.109416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2672.451397] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 18:59:14 (1784415554) [ 2672.784592] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2672.785995] LustreError: 6487:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b5ff5a2d80 x1871092591100928/t266287972368(0) o36->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:290/0 lens 504/448 e 0 to 0 dl 1784415610 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2674.051215] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2676.716650] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2677.337854] Lustre: Failing over lustre-MDT0000 [ 2677.435729] Lustre: server umount lustre-MDT0000 complete [ 2692.461452] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2692.463066] LDISKFS-fs (dm-0): recovery complete [ 2692.466430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.251351] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2697.716212] Lustre: 6486:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5ef40e680 x1871092591100928/t266287972368(0) o36->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:315/0 lens 504/2880 e 0 to 0 dl 1784415635 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2697.726908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 2697.726913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3297) [ 2697.728326] Lustre: 6486:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 2702.983958] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 18:59:44 (1784415584) [ 2703.429420] LustreError: 6487:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b5ff959500 x1871092591113216/t270582939664(0) o36->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:321/0 lens 504/448 e 0 to 0 dl 1784415641 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2703.444228] LustreError: 6487:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 2704.849456] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2704.852687] Lustre: Skipped 1 previous similar message [ 2708.109823] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2708.715548] Lustre: Failing over lustre-MDT0000 [ 2708.800364] Lustre: server umount lustre-MDT0000 complete [ 2723.732832] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2723.735478] LDISKFS-fs (dm-0): recovery complete [ 2723.739180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2725.426860] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2728.949273] Lustre: 12413:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5ff885500 x1871092591113216/t270582939664(0) o36->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:346/0 lens 504/2880 e 0 to 0 dl 1784415666 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2728.957920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 2728.957991] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3297) [ 2733.069747] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 19:00:15 (1784415615) [ 2733.470529] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2734.888309] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2734.892366] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2734.896022] Lustre: Skipped 2 previous similar messages [ 2738.932467] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2739.783549] Lustre: Failing over lustre-MDT0000 [ 2739.913876] Lustre: server umount lustre-MDT0000 complete [ 2755.023871] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2755.026716] LDISKFS-fs (dm-0): recovery complete [ 2755.031238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2756.891743] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2760.178286] Lustre: 6488:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5f55f9f80 x1871092591125632/t274877906960(0) o35->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:333/0 lens 392/456 e 0 to 0 dl 1784415653 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2760.186905] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 2760.186913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3361) [ 2764.679375] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 19:00:46 (1784415646) [ 2765.025516] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 2765.028173] LustreError: 7435:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b4c7d79500 x1871092591136512/t279172874255(0) o101->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:382/0 lens 664/608 e 0 to 0 dl 1784415702 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2765.036875] LustreError: 7435:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 2825.133263] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnecting [ 2825.135763] Lustre: Skipped 4 previous similar messages [ 2825.141577] Lustre: 6487:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5ff919180 x1871092591136512/t279172874255(0) o101->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:442/0 lens 664/3488 e 0 to 0 dl 1784415762 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2827.448036] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 19:01:49 (1784415709) [ 2829.896653] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2830.538539] Lustre: Failing over lustre-MDT0000 [ 2830.638603] Lustre: server umount lustre-MDT0000 complete [ 2831.839682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.840085] LustreError: 6486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2831.843074] Lustre: Skipped 58 previous similar messages [ 2831.849700] LustreError: 6486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 171 previous similar messages [ 2844.446955] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2844.448586] LDISKFS-fs (dm-0): recovery complete [ 2844.452284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.554654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2844.557897] Lustre: Skipped 13 previous similar messages [ 2845.824459] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2849.762150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2849.764116] Lustre: Skipped 55 previous similar messages [ 2849.771887] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2849.774684] Lustre: Skipped 12 previous similar messages [ 2849.788302] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 2849.788317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3393) [ 2851.396064] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2851.854834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2865.206700] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 19:02:27 (1784415747) [ 2867.781784] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2868.370991] Lustre: Failing over lustre-MDT0000 [ 2868.615887] Lustre: server umount lustre-MDT0000 complete [ 2882.436354] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2882.437469] LDISKFS-fs (dm-0): recovery complete [ 2882.440464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.473433] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2882.476875] LustreError: Skipped 11 previous similar messages [ 2883.818479] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2887.672307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 2887.672315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3393) [ 2889.305815] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2889.827845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2892.201634] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2897.183240] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 19:02:59 (1784415779) [ 2906.060914] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2906.684292] Lustre: Failing over lustre-MDT0000 [ 2907.064224] Lustre: server umount lustre-MDT0000 complete [ 2920.928323] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2920.930369] LDISKFS-fs (dm-0): recovery complete [ 2920.935024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.462143] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2926.687065] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 2926.687065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 2928.439281] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2928.910841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2942.089666] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 19:03:44 (1784415824) [ 2945.912228] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2946.692974] Lustre: Failing over lustre-MDT0000 [ 2946.861068] Lustre: server umount lustre-MDT0000 complete [ 2960.624099] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2960.625659] LDISKFS-fs (dm-0): recovery complete [ 2960.628599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.987150] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2966.012497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 2966.012509] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 2967.677554] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2968.131947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2971.358503] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 2971.811728] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2973.720521] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 19:04:15 (1784415855) [ 2979.477643] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3040.957869] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3040.959644] Lustre: Skipped 1 previous similar message [ 3040.961042] LustreError: 12413:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b5d8266c50 x1871092593924224/t296352743435(0) o36->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:658/0 lens 66040/440 e 0 to 0 dl 1784415978 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3100.582133] Lustre: 7435:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b4c7c10700 x1871092593924224/t296352743435(0) o36->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:718/0 lens 66040/440 e 0 to 0 dl 1784416038 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3103.308968] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3103.789458] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 19:06:25 (1784415985) [ 3107.713703] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3108.449152] Lustre: Failing over lustre-MDT0000 [ 3108.555822] Lustre: server umount lustre-MDT0000 complete [ 3109.344174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3109.347185] LustreError: Skipped 11 previous similar messages [ 3122.499518] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3122.501540] LDISKFS-fs (dm-0): recovery complete [ 3122.504843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.002127] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3126.180594] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3126.184081] Lustre: Skipped 13 previous similar messages [ 3127.952593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 3127.952615] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 3129.744451] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3130.244530] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3133.828516] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 19:06:55 (1784416015) [ 3139.105219] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3141.934166] Lustre: Failing over lustre-OST0000 [ 3141.977429] Lustre: server umount lustre-OST0000 complete [ 3155.925057] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3155.926590] LDISKFS-fs (dm-2): recovery complete [ 3155.929163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3155.981144] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3155.983200] Lustre: Skipped 12 previous similar messages [ 3157.972620] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3169.404130] Lustre: Failing over lustre-OST0000 [ 3169.449365] Lustre: server umount lustre-OST0000 complete [ 3182.566382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3184.777829] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3188.510969] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3189.019130] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3223.228297] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 19:08:25 (1784416105) [ 3224.161057] Lustre: Failing over lustre-MDT0000 [ 3224.440285] Lustre: server umount lustre-MDT0000 complete [ 3237.494767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.143944] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3243.001686] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 3243.001689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 3250.687725] Lustre: Failing over lustre-MDT0000 [ 3250.896580] Lustre: server umount lustre-MDT0000 complete [ 3263.961761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.657503] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3269.636735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 3269.636802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 3271.716228] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3272.212276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3275.883761] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 19:09:17 (1784416157) [ 3287.718675] Lustre: Failing over lustre-OST0000 [ 3287.781121] Lustre: server umount lustre-OST0000 complete [ 3301.168932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3303.281060] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3307.211587] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3307.760219] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3311.454597] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 19:09:53 (1784416193) [ 3312.301783] Lustre: Failing over lustre-MDT0000 [ 3312.567432] Lustre: server umount lustre-MDT0000 complete [ 3316.208949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.247853] Lustre: *** cfs_fail_loc=605, val=0*** [ 3316.249130] LustreError: 164158:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc08d2100 failed: rc = -95 [ 3316.251912] LustreError: 164158:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 3316.254103] LustreError: 164158:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 3316.256467] LustreError: 164158:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 3316.258761] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3316.260922] LustreError: 164158:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3316.265646] Lustre: server umount lustre-MDT0000 complete [ 3316.266914] LustreError: 164158:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3319.039497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.513336] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3323.090870] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 19:10:05 (1784416205) [ 3324.411811] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3324.411816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3326.806149] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3327.733214] Lustre: Failing over lustre-MDT0000 [ 3327.967983] Lustre: server umount lustre-MDT0000 complete [ 3343.004047] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3343.007211] LDISKFS-fs (dm-0): recovery complete [ 3343.011549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3344.845814] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3346.344177] Lustre: *** cfs_fail_loc=707, val=0*** [ 3405.747519] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 3405.772384] Lustre: 166327:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3405.775923] Lustre: 166327:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 1 previous similar message [ 3405.950093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 3405.950107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 3408.569632] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3409.373090] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3414.165619] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 19:11:36 (1784416296) [ 3438.206578] LustreError: 20278:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff95b5e590ad80 x1871092594803072/t0(0) o101->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:256/0 lens 664/0 e 0 to 0 dl 1784416331 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3438.217492] LustreError: 20278:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3449.247131] LustreError: 20278:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3449.254695] LustreError: 38536:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff95b4c19bf800 x1871092594804736/t0(0) o35->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:267/0 lens 392/0 e 0 to 0 dl 1784416342 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3454.944566] LustreError: 17082:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff95b5ff729180 x1871092599113984/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:273/0 lens 544/0 e 0 to 0 dl 1784416348 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3454.951514] LustreError: 17082:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 3461.140182] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 19:12:23 (1784416343) [ 3484.274599] LustreError: 42608:0:(tgt_handler.c:2836:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3495.303085] LustreError: 42608:0:(tgt_handler.c:2836:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3498.078518] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 19:13:00 (1784416380) [ 3520.817748] LustreError: 6485:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff95b5f51b9f80 x1871092594828416/t0(0) o101->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:339/0 lens 576/0 e 0 to 0 dl 1784416414 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3520.825049] LustreError: 6485:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3525.927068] LustreError: 6485:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3525.930869] LustreError: 117305:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff95b5f59fb480 x1871092594829312/t0(0) o101->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:344/0 lens 664/0 e 0 to 0 dl 1784416419 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3526.615294] LustreError: 6485:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 10000ms [ 3536.703092] LustreError: 6485:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3536.712419] LustreError: 20278:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff95b5ff755180 x1871092594851072/t0(0) o101->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:394/0 lens 664/0 e 0 to 0 dl 1784416469 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3536.730803] LustreError: 20278:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 3548.484780] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 19:13:50 (1784416430) [ 3634.043339] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 19:15:16 (1784416516) [ 3657.006917] LustreError: 6487:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff95b6000e5c00 x1871092594907520/t0(0) o101->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:475/0 lens 576/0 e 0 to 0 dl 1784416550 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3657.014577] LustreError: 6487:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 3657.017635] LustreError: 6487:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3657.439093] LustreError: 6487:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3673.313953] LustreError: 7435:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3673.320251] LustreError: 7435:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3673.743096] LustreError: 7435:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3673.748660] LustreError: 7435:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3689.178090] LustreError: 20278:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff95b5e7a81c00 x1871092594927104/t0(0) o36->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:507/0 lens 504/0 e 0 to 0 dl 1784416582 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 3689.193546] LustreError: 20278:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 80 previous similar messages [ 3701.834625] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 19:16:24 (1784416584) [ 3726.760431] Lustre: DEBUG MARKER: phase 2 [ 3730.262243] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 19:16:52 (1784416612) [ 3802.422550] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 19:18:04 (1784416684) [ 3803.133844] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 3803.907125] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 19:18:05 (1784416685) [ 3805.883305] Lustre: DEBUG MARKER: Started rundbench load pid=134891 ... [ 3809.661803] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3811.366061] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 3812.091286] Lustre: Failing over lustre-MDT0000 [ 3812.313700] Lustre: server umount lustre-MDT0000 complete [ 3813.977059] LustreError: 117305:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3813.981380] LustreError: 117305:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 145 previous similar messages [ 3814.368556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3814.377132] Lustre: Skipped 40 previous similar messages [ 3827.170300] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3827.171709] LDISKFS-fs (dm-0): recovery complete [ 3827.175671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3827.222401] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3827.226322] LustreError: Skipped 7 previous similar messages [ 3827.301493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3827.303358] Lustre: Skipped 11 previous similar messages [ 3827.314422] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3827.317167] Lustre: Skipped 6 previous similar messages [ 3827.621696] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3827.624242] Lustre: Skipped 7 previous similar messages [ 3828.727369] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3832.807410] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3832.812659] Lustre: Skipped 41 previous similar messages [ 3832.820438] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3832.823060] Lustre: Skipped 11 previous similar messages [ 3832.838927] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 3832.839076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 3834.927215] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3835.524389] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3840.960156] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3842.550308] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 3843.357718] Lustre: Failing over lustre-MDT0001 [ 3843.366850] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.38@tcp (stopping) [ 3843.550115] Lustre: server umount lustre-MDT0001 complete [ 3847.647724] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3847.652119] LustreError: Skipped 5 previous similar messages [ 3858.869061] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 3858.870436] LDISKFS-fs (dm-1): recovery complete [ 3858.874840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3860.199863] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3864.403825] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:703 to 0x2c0000400:737) [ 3864.403989] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:720 to 0x280000400:737) [ 3866.466069] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3867.064474] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3872.254660] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3873.820810] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 3874.588814] Lustre: Failing over lustre-MDT0000 [ 3874.864621] Lustre: server umount lustre-MDT0000 complete [ 3889.225326] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3889.227587] LDISKFS-fs (dm-0): recovery complete [ 3889.232472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.946873] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3894.787223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 3894.787255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 3896.496872] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3896.962078] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3902.042369] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3903.876322] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 3904.610912] Lustre: Failing over lustre-MDT0001 [ 3904.652045] LustreError: 7435:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000ff519440 ns: mdt-lustre-MDT0001_UUID lock: ffff95b600247a00/0x82f3fcd33f7a9ea9 lrc: 3/0,0 mode: PR/PR res: [0x2400007ed:0x86f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.203.38@tcp remote: 0xf335048dbd5f2178 expref: 3 pid: 7435 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3904.773800] Lustre: server umount lustre-MDT0001 complete [ 3919.789295] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 3919.790613] LDISKFS-fs (dm-1): recovery complete [ 3919.795355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3921.407180] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3925.359889] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1101 to 0x280000400:1121) [ 3925.359916] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1101 to 0x2c0000400:1121) [ 3927.513189] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3928.137362] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3952.309720] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 19:20:34 (1784416834) [ 4075.543596] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4086.366592] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 4087.139895] Lustre: Failing over lustre-MDT0001 [ 4087.178132] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.38@tcp (stopping) [ 4087.438303] Lustre: server umount lustre-MDT0001 complete [ 4102.636649] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4102.638645] LDISKFS-fs (dm-1): recovery complete [ 4102.644096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.218306] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4110.997262] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2269 to 0x2c0000400:2305) [ 4110.997580] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2269 to 0x280000400:2305) [ 4112.940226] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4113.480408] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4238.792386] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4249.773786] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 4250.731663] Lustre: Failing over lustre-MDT0000 [ 4250.754127] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.38@tcp (stopping) [ 4250.991979] Lustre: server umount lustre-MDT0000 complete [ 4266.790594] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4266.793586] LDISKFS-fs (dm-0): recovery complete [ 4266.800556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4268.882496] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4274.835280] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6845 to 0x2c0000401:6881) [ 4274.837259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6878 to 0x280000401:6913) [ 4277.462225] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4278.215987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4316.229795] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 19:26:38 (1784417198) [ 4439.437913] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4450.302682] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 4451.040944] Lustre: Failing over lustre-MDT0000 [ 4451.065956] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.38@tcp (stopping) [ 4451.295718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4451.300855] Lustre: Skipped 21 previous similar messages [ 4456.415662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4456.419481] Lustre: Skipped 8 previous similar messages [ 4457.076293] Lustre: server umount lustre-MDT0000 complete [ 4457.380855] LustreError: 12413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.397373] LustreError: 12413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 78 previous similar messages [ 4472.591591] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4472.594990] LDISKFS-fs (dm-0): recovery complete [ 4472.600372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.653577] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4472.656157] LustreError: Skipped 2 previous similar messages [ 4472.738339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.742826] Lustre: Skipped 5 previous similar messages [ 4472.752809] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4472.754510] Lustre: Skipped 5 previous similar messages [ 4473.832539] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4473.834743] Lustre: Skipped 5 previous similar messages [ 4473.907955] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4477.923142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4477.927160] Lustre: Skipped 20 previous similar messages [ 4479.397038] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4479.401913] Lustre: Skipped 5 previous similar messages [ 4479.421848] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7288 to 0x2c0000401:7329) [ 4479.421879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7320 to 0x280000401:7361) [ 4481.673897] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4482.440552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4607.969348] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4619.042544] Lustre: DEBUG MARKER: test_70d fail mds2 2 times [ 4620.003803] Lustre: Failing over lustre-MDT0001 [ 4620.003997] Lustre: 12413:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 0/real 0] req@ffff95b5f55b3b80 x1871092617507968/t0(0) o104->lustre-MDT0001@0@lo:15/16 lens 328/224 e 0 to 1 dl 0 ref 1 fl Rpc:EeXQU/0/ffffffff rc -19/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 [ 4620.028185] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.38@tcp (stopping) [ 4620.296570] Lustre: server umount lustre-MDT0001 complete [ 4621.281435] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4621.287157] LustreError: Skipped 3 previous similar messages [ 4636.065155] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4636.067721] LDISKFS-fs (dm-1): recovery complete [ 4636.074483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.865803] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4642.402104] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3736 to 0x2c0000400:3777) [ 4642.402252] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3736 to 0x280000400:3777) [ 4645.226446] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4645.977368] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4650.780051] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 19:32:12 (1784417532) [ 4775.209806] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4786.220960] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 4787.175851] Lustre: Failing over lustre-MDT0000 [ 4787.189161] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.38@tcp (stopping) [ 4787.480724] Lustre: server umount lustre-MDT0000 complete [ 4803.221761] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4803.224455] LDISKFS-fs (dm-0): recovery complete [ 4803.229723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4805.060949] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4811.936811] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8816 to 0x2c0000401:8833) [ 4811.947774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8848 to 0x280000401:8865) [ 4814.388874] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4814.987940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4940.581891] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4951.614572] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 4952.532571] Lustre: Failing over lustre-MDT0000 [ 4952.892701] Lustre: server umount lustre-MDT0000 complete [ 4968.273900] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4968.276233] LDISKFS-fs (dm-0): recovery complete [ 4968.280824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.052370] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4977.219829] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10123 to 0x2c0000401:10145) [ 4977.220236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10155 to 0x280000401:10177) [ 4979.679853] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4980.277362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4984.329819] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 19:37:46 (1784417866) [ 4991.282970] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4993.074522] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 4993.982584] Lustre: Failing over lustre-OST0000 [ 4994.049612] Lustre: server umount lustre-OST0000 complete [ 5009.303987] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5009.306561] LDISKFS-fs (dm-2): recovery complete [ 5009.313932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5011.930869] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5016.430805] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5017.026176] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5027.028911] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5028.816687] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5029.724220] Lustre: Failing over lustre-OST0000 [ 5029.780699] Lustre: server umount lustre-OST0000 complete [ 5044.864750] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5044.866032] LDISKFS-fs (dm-2): recovery complete [ 5044.869302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5047.251931] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5052.184499] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5052.853336] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5059.731128] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 19:39:01 (1784417941) [ 5183.715629] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5186.921578] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5197.678644] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 5198.601489] Lustre: Failing over lustre-MDT0000 [ 5198.673809] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.38@tcp (stopping) [ 5198.817021] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.820606] Lustre: Skipped 17 previous similar messages [ 5198.915603] Lustre: server umount lustre-MDT0000 complete [ 5200.509815] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418083 with bad export cookie 9436163628509208947 [ 5200.510538] Lustre: Failing over lustre-MDT0001 [ 5200.511138] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5200.511142] LustreError: Skipped 2 previous similar messages [ 5200.513644] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5200.792204] Lustre: server umount lustre-MDT0001 complete [ 5216.768882] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5216.770235] LDISKFS-fs (dm-0): recovery complete [ 5216.773843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.789723] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5216.792560] LDISKFS-fs (dm-1): recovery complete [ 5216.798091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5220.319092] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784418086/real 1784418086] req@ffff95b600f7bb80 x1871092625248896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784418102 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5220.339174] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5222.367235] Lustre: 3620:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784418088/real 1784418088] req@ffff95b4ce5a2680 x1871092625249024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784418104 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5222.382145] Lustre: 3620:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5224.415151] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784418091/real 1784418091] req@ffff95b600f7b800 x1871092625249280/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784418107 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5225.503468] LustreError: 3617:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95b4c63dbb80 x1871092625251072/t0(0) o250->MGC192.168.203.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5225.651992] LustreError: 180137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5225.660686] LustreError: 180137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 75 previous similar messages [ 5225.676518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5225.681665] Lustre: Skipped 5 previous similar messages [ 5225.697652] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5225.700834] Lustre: Skipped 5 previous similar messages [ 5225.782130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 5225.786213] LustreError: Skipped 2 previous similar messages [ 5227.200289] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5227.324805] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5227.365041] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5227.366919] Lustre: Skipped 5 previous similar messages [ 5229.921158] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5229.922772] Lustre: Skipped 18 previous similar messages [ 5230.950323] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5230.952507] Lustre: Skipped 5 previous similar messages [ 5230.965096] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3788 to 0x280000400:3809) [ 5230.965095] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3788 to 0x2c0000400:3809) [ 5231.967085] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784418098/real 1784418098] req@ffff95b4ce5a0380 x1871092625249920/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784418114 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5231.973241] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5239.177609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10244 to 0x280000401:10273) [ 5239.177975] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10213 to 0x2c0000401:10241) [ 5241.663034] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5242.242592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5242.831486] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5245.407186] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784418102/real 1784418102] req@ffff95b600f7aa00 x1871092625250816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784418118 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5245.423812] Lustre: 3618:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5247.118152] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 19:42:09 (1784418129) [ 5250.388038] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5251.569838] Lustre: Failing over lustre-MDT0000 [ 5251.807765] Lustre: server umount lustre-MDT0000 complete [ 5267.333709] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5267.336271] LDISKFS-fs (dm-0): recovery complete [ 5267.342963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5268.997039] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5271.463586] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5287.851198] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5287.902936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10244 to 0x280000401:10305) [ 5287.902953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10243 to 0x2c0000401:10273) [ 5289.994593] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5290.624923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5294.807520] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 19:42:56 (1784418176) [ 5298.157846] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5299.395400] Lustre: Failing over lustre-MDT0000 [ 5299.627915] Lustre: server umount lustre-MDT0000 complete [ 5314.369326] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5314.371308] LDISKFS-fs (dm-0): recovery complete [ 5314.376279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5316.107978] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5317.554446] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5317.557184] LustreError: 198589:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b4c19bea00 x1871092656851968/t343597384359(343597384359) o101->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:626/0 lens 520/664 e 0 to 0 dl 1784418211 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5333.939970] Lustre: lustre-MDT0000: Client c2195709-2ef6-4414-8a03-c1f4c705ea9b (at 192.168.203.38@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5333.945973] Lustre: 200852:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5ef40df80 x1871092656851968/t343597384359(343597384359) o101->c2195709-2ef6-4414-8a03-c1f4c705ea9b@192.168.203.38@tcp:642/0 lens 520/3488 e 0 to 0 dl 1784418227 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5333.971819] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10243 to 0x2c0000401:10305) [ 5333.971825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10337) [ 5336.321252] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5336.890357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5340.875164] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 19:43:42 (1784418222) [ 5342.292978] Lustre: Failing over lustre-OST0000 [ 5342.343212] Lustre: server umount lustre-OST0000 complete [ 5343.686336] Lustre: Failing over lustre-MDT0000 [ 5343.802959] Lustre: server umount lustre-MDT0000 complete [ 5357.128646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.799655] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5359.747770] Lustre: lustre-MDT0000: Denying connection for new client cefe474c-7752-4514-aca7-3214b32ed13e (at 192.168.203.38@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5362.687951] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10243 to 0x2c0000401:10337) [ 5367.451563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5369.196780] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5372.816358] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 19:44:15 (1784418255) [ 5372.902224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10369) [ 5373.123325] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5373.124611] LustreError: 8408:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b4cf9f6680 x1871092626474240/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:681/0 lens 1312/4320 e 0 to 0 dl 1784418266 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5375.625550] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5376.467293] Lustre: Failing over lustre-MDT0000 [ 5376.751202] Lustre: server umount lustre-MDT0000 complete [ 5389.279194] Lustre: 198067:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784418255/real 1784418255] req@ffff95b4cf9f5500 x1871092626474240/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1784418271 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5392.263379] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5392.265957] LDISKFS-fs (dm-0): recovery complete [ 5392.271902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.997518] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5397.511980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10401) [ 5397.512325] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10369) [ 5399.758471] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5400.385838] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5404.379350] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 19:44:46 (1784418286) [ 5407.307406] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5410.037770] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5410.854608] Lustre: Failing over lustre-MDT0001 [ 5410.862819] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.38@tcp (stopping) [ 5410.867224] Lustre: Skipped 4 previous similar messages [ 5417.058691] Lustre: server umount lustre-MDT0001 complete [ 5432.131342] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5432.133210] LDISKFS-fs (dm-1): recovery complete [ 5432.138181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.879204] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5437.427980] Lustre: 198589:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b600f79880 x1871092661743104/t25769805823(0) o36->cefe474c-7752-4514-aca7-3214b32ed13e@192.168.203.38@tcp:16/0 lens 560/2880 e 0 to 0 dl 1784418356 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5437.441451] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3820 to 0x280000400:3841) [ 5437.441482] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3820 to 0x2c0000400:3841) [ 5439.575536] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5440.189346] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5443.979809] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 19:45:26 (1784418326) [ 5444.429843] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5444.432933] Lustre: Skipped 1 previous similar message [ 5444.435493] LustreError: 180138:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b5e578df80 x1871092626537344/t365072220193(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:753/0 lens 2520/4320 e 0 to 0 dl 1784418338 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5444.448683] LustreError: 180138:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 5447.414080] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5450.471049] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5451.296268] Lustre: Failing over lustre-MDT0000 [ 5451.584290] Lustre: server umount lustre-MDT0000 complete [ 5460.959322] Lustre: 208894:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784418327/real 1784418327] req@ffff95b5e578f800 x1871092626537344/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1784418343 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5466.970555] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5466.972892] LDISKFS-fs (dm-0): recovery complete [ 5466.977058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.433165] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5472.295681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10433) [ 5472.295689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10401) [ 5473.999523] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5474.489439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5476.425055] Lustre: Failing over lustre-MDT0001 [ 5476.512994] Lustre: server umount lustre-MDT0001 complete [ 5491.066342] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5491.069014] LDISKFS-fs (dm-1): recovery complete [ 5491.076149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5492.638576] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5496.318961] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3852 to 0x280000400:3873) [ 5496.318968] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3852 to 0x2c0000400:3873) [ 5498.029787] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5498.530526] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5502.451923] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 19:46:24 (1784418384) [ 5508.579183] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5511.473387] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5512.198321] Lustre: Failing over lustre-MDT0000 [ 5512.482479] Lustre: server umount lustre-MDT0000 complete [ 5514.012165] LustreError: 8414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418396 with bad export cookie 9436163628509944794 [ 5514.014195] Lustre: Failing over lustre-MDT0001 [ 5514.019220] LustreError: 8414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5519.839780] LustreError: 214533:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff95b4c656df80 x1871092626600192/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5519.850807] LustreError: 214533:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5519.981576] Lustre: server umount lustre-MDT0001 complete [ 5535.326303] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5535.329287] LDISKFS-fs (dm-0): recovery complete [ 5535.333093] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5535.333921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.334780] LDISKFS-fs (dm-1): recovery complete [ 5535.343207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.434119] LustreError: 215632:0:(llog.c:1655:llog_backup()) MGC192.168.203.138@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5535.437621] Lustre: 215632:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.138@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5539.295527] LustreError: 3617:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95b5ff4ff100 x1871092626601344/t0(0) o250->MGC192.168.203.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5540.829083] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5540.941648] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5544.580986] Lustre: 215645:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b600cd2a00 x1871092661811840/t34359738436(0) o36->cefe474c-7752-4514-aca7-3214b32ed13e@192.168.203.38@tcp:123/0 lens 560/2880 e 0 to 0 dl 1784418463 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5544.586813] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3884 to 0x2c0000400:3905) [ 5544.596254] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3884 to 0x280000400:3905) [ 5544.612909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10465) [ 5544.613398] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10433) [ 5547.003193] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5547.580054] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5548.222275] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5552.571698] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 19:47:14 (1784418434) [ 5553.036126] LustreError: 216422:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b4c19bdf80 x1871092661845120/t38654705734(0) o36->cefe474c-7752-4514-aca7-3214b32ed13e@192.168.203.38@tcp:145/0 lens 560/448 e 0 to 0 dl 1784418485 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5553.050555] LustreError: 216422:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 5559.336435] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5560.167705] Lustre: Failing over lustre-MDT0000 [ 5560.287654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5560.290219] Lustre: Skipped 11 previous similar messages [ 5560.459247] Lustre: server umount lustre-MDT0000 complete [ 5576.104445] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5576.106991] LDISKFS-fs (dm-0): recovery complete [ 5576.111527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5578.094386] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5581.317179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10497) [ 5581.317219] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10465) [ 5583.839889] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5584.509034] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5608.357203] Lustre: lustre-MDT0001: Client cefe474c-7752-4514-aca7-3214b32ed13e (at 192.168.203.38@tcp) reconnecting [ 5608.363559] Lustre: Skipped 2 previous similar messages [ 5608.368626] Lustre: 216422:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5ff40a300 x1871092661845120/t38654705734(0) o36->cefe474c-7752-4514-aca7-3214b32ed13e@192.168.203.38@tcp:201/0 lens 560/2880 e 0 to 0 dl 1784418541 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5611.967855] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 19:48:13 (1784418493) [ 5612.496459] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5612.501181] Lustre: Skipped 2 previous similar messages [ 5616.056411] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5617.020551] Lustre: Failing over lustre-MDT0001 [ 5617.148104] Lustre: server umount lustre-MDT0001 complete [ 5632.056598] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5632.059092] LDISKFS-fs (dm-1): recovery complete [ 5632.065485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.055190] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5637.637130] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3926 to 0x2c0000400:3969) [ 5637.637418] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3926 to 0x280000400:3969) [ 5640.196283] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5640.820213] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5645.274727] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 19:48:47 (1784418527) [ 5651.680749] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5655.304563] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5656.250501] Lustre: Failing over lustre-MDT0000 [ 5656.492401] Lustre: server umount lustre-MDT0000 complete [ 5662.119415] Lustre: 220932:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5d1953800 x1871092661910912/t42949673030(0) o36->cefe474c-7752-4514-aca7-3214b32ed13e@192.168.203.38@tcp:215/0 lens 560/2880 e 0 to 0 dl 1784418555 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5662.133372] Lustre: 220932:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5672.222249] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5672.223688] LDISKFS-fs (dm-0): recovery complete [ 5672.227076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.087177] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5677.590558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10529) [ 5677.590670] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10497) [ 5680.066159] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5680.855891] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5683.054072] Lustre: Failing over lustre-MDT0001 [ 5683.140023] Lustre: server umount lustre-MDT0001 complete [ 5698.398405] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5698.401350] LDISKFS-fs (dm-1): recovery complete [ 5698.410086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5700.044980] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5703.671449] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3980 to 0x2c0000400:4001) [ 5703.671449] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3980 to 0x280000400:4001) [ 5705.696371] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5706.499639] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5710.802071] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 19:49:52 (1784418592) [ 5711.284662] LustreError: 220932:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b5d17c4700 x1871092661949824/t47244640326(0) o36->cefe474c-7752-4514-aca7-3214b32ed13e@192.168.203.38@tcp:264/0 lens 560/448 e 0 to 0 dl 1784418604 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5711.298110] LustreError: 220932:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 2 previous similar messages [ 5717.730366] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5720.775607] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5721.598829] Lustre: Failing over lustre-MDT0000 [ 5721.858866] Lustre: server umount lustre-MDT0000 complete [ 5723.389577] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418606 with bad export cookie 9436163628509958584 [ 5723.390833] Lustre: Failing over lustre-MDT0001 [ 5723.392421] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5723.484092] Lustre: server umount lustre-MDT0001 complete [ 5739.295416] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5739.299094] LDISKFS-fs (dm-0): recovery complete [ 5739.301474] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5739.304056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.304928] LDISKFS-fs (dm-1): recovery complete [ 5739.316526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.404177] LustreError: 226807:0:(llog.c:1655:llog_backup()) MGC192.168.203.138@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5739.406931] Lustre: 226807:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.138@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5739.999171] Lustre: 3621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784418606/real 1784418606] req@ffff95b5dbfd6680 x1871092626768128/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784418622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5740.021218] Lustre: 3621:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5749.215685] LustreError: 3617:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95b5ff655f80 x1871092626771712/t0(0) o250->MGC192.168.203.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5750.796290] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5750.918937] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5755.501950] Lustre: 226820:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5d1951c00 x1871092661949824/t47244640326(0) o36->cefe474c-7752-4514-aca7-3214b32ed13e@192.168.203.38@tcp:309/0 lens 560/2880 e 0 to 0 dl 1784418649 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5755.532206] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4012 to 0x2c0000400:4033) [ 5755.532210] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4012 to 0x280000400:4033) [ 5755.545401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10561) [ 5755.545445] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10529) [ 5757.376410] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5758.146910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5758.794859] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5763.467492] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 19:50:45 (1784418645) [ 5767.462243] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5768.380302] Lustre: Failing over lustre-MDT0001 [ 5768.385223] LustreError: 228616:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x240002ef9:0xa9:0x0].0x0 (ffff95b60022b600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5773.896289] Lustre: server umount lustre-MDT0001 complete [ 5789.606775] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5789.609298] LDISKFS-fs (dm-1): recovery complete [ 5789.617722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.507470] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5794.802731] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4044 to 0x280000400:4065) [ 5794.802757] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4044 to 0x2c0000400:4065) [ 5796.460796] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5797.114846] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5801.045183] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 19:51:23 (1784418683) [ 5803.548588] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5804.093850] Lustre: Failing over lustre-MDT0000 [ 5804.325107] Lustre: server umount lustre-MDT0000 complete [ 5805.023804] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5805.026861] Lustre: Skipped 57 previous similar messages [ 5819.691404] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5819.693097] LDISKFS-fs (dm-0): recovery complete [ 5819.696830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5819.741844] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5819.744888] LustreError: Skipped 9 previous similar messages [ 5821.115973] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5825.040862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10593) [ 5825.040917] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10561) [ 5826.861393] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5827.325165] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5830.450113] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 19:51:52 (1784418712) [ 5833.219272] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5836.656519] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5837.553990] Lustre: Failing over lustre-MDT0000 [ 5837.734140] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.38@tcp (stopping) [ 5837.737527] Lustre: Skipped 5 previous similar messages [ 5837.910726] Lustre: server umount lustre-MDT0000 complete [ 5840.352707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5840.353285] LustreError: 180138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5840.357951] LustreError: Skipped 15 previous similar messages [ 5840.367751] LustreError: 180138:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 213 previous similar messages [ 5853.645536] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5853.648333] LDISKFS-fs (dm-0): recovery complete [ 5853.654273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.803907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5853.809116] Lustre: Skipped 19 previous similar messages [ 5853.857310] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5853.859109] Lustre: Skipped 19 previous similar messages [ 5855.170955] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5858.212906] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5858.217127] Lustre: Skipped 19 previous similar messages [ 5859.312349] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5859.312934] Lustre: 233609:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5859.317959] Lustre: Skipped 65 previous similar messages [ 5859.328355] Lustre: 233609:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 53 previous similar messages [ 5859.348051] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5859.356180] Lustre: Skipped 19 previous similar messages [ 5859.388580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10625) [ 5859.388601] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10593) [ 5861.813874] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5862.590970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5864.960619] Lustre: Failing over lustre-MDT0001 [ 5865.063477] Lustre: server umount lustre-MDT0001 complete [ 5880.852164] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5880.854838] LDISKFS-fs (dm-1): recovery complete [ 5880.861842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5882.747785] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5886.467450] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4044 to 0x2c0000400:4097) [ 5886.467466] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4044 to 0x280000400:4097) [ 5888.923378] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5889.716737] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5893.840359] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 19:52:55 (1784418775) [ 5894.415418] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5894.418393] Lustre: Skipped 5 previous similar messages [ 5898.140949] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5901.834617] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5902.835369] Lustre: Failing over lustre-MDT0000 [ 5903.143025] Lustre: server umount lustre-MDT0000 complete [ 5904.908172] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418787 with bad export cookie 9436163628509969035 [ 5904.910203] Lustre: Failing over lustre-MDT0001 [ 5904.915875] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5904.920069] LustreError: 236862:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029441:0x1f:0x0].0x0 (ffff95b5ef4b3300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5904.933707] LustreError: 236862:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message [ 5911.007813] LustreError: 236862:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff95b4c67b0380 x1871092626919936/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5911.018299] LustreError: 236862:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5911.163132] Lustre: server umount lustre-MDT0001 complete [ 5927.282178] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5927.283227] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5927.284007] LDISKFS-fs (dm-1): recovery complete [ 5927.285068] LDISKFS-fs (dm-0): recovery complete [ 5927.289615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5927.289930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5927.370169] LustreError: 237961:0:(llog.c:1655:llog_backup()) MGC192.168.203.138@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5927.373773] Lustre: 237961:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.138@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5929.439714] LustreError: 3617:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95b600242300 x1871092626921088/t0(0) o250->MGC192.168.203.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5931.097766] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5931.185751] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5935.751955] Lustre: 237970:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95b5e5837480 x1871092662030080/t60129542146(0) o36->cefe474c-7752-4514-aca7-3214b32ed13e@192.168.203.38@tcp:514/0 lens 496/2888 e 0 to 0 dl 1784418854 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 5935.760924] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4044 to 0x280000400:4129) [ 5935.760961] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4044 to 0x2c0000400:4129) [ 5935.765909] Lustre: 237970:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5935.812678] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10625) [ 5935.812685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10657) [ 5938.389681] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5939.127866] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5939.841634] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5944.257829] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 19:53:46 (1784418826) [ 5948.597917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5949.533269] Lustre: Failing over lustre-MDT0000 [ 5949.713816] Lustre: server umount lustre-MDT0000 complete [ 5961.128280] Lustre: lustre-MDT0001: Client cefe474c-7752-4514-aca7-3214b32ed13e (at 192.168.203.38@tcp) reconnecting [ 5961.133743] Lustre: Skipped 1 previous similar message [ 5965.107797] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5965.110714] LDISKFS-fs (dm-0): recovery complete [ 5965.116080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.987367] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5970.454183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10689) [ 5970.454203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10657) [ 5972.975336] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5973.727929] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5978.111616] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 19:54:20 (1784418860) [ 5978.574594] LustreError: 237971:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff95b600f79880 x1871092662057984/t64424509445(0) o36->cefe474c-7752-4514-aca7-3214b32ed13e@192.168.203.38@tcp:532/0 lens 496/456 e 0 to 0 dl 1784418872 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 5978.599170] LustreError: 237971:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 5 previous similar messages [ 5982.094824] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5982.929093] Lustre: Failing over lustre-MDT0001 [ 5983.002136] Lustre: server umount lustre-MDT0001 complete [ 5998.351842] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5998.353456] LDISKFS-fs (dm-1): recovery complete [ 5998.358142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6000.013433] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6003.711187] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4044 to 0x280000400:4161) [ 6003.711230] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4044 to 0x2c0000400:4161) [ 6006.119245] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6006.873829] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6011.263398] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 19:54:53 (1784418893) [ 6015.077248] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6018.618126] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6019.446263] Lustre: Failing over lustre-MDT0000 [ 6019.596169] Lustre: server umount lustre-MDT0000 complete [ 6033.725211] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6033.727825] LDISKFS-fs (dm-0): recovery complete [ 6033.732900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6035.068768] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6039.036739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10721) [ 6039.036744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10689) [ 6040.907486] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6041.616969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6043.945924] Lustre: Failing over lustre-MDT0001 [ 6044.061852] Lustre: server umount lustre-MDT0001 complete [ 6059.820145] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6059.822817] LDISKFS-fs (dm-1): recovery complete [ 6059.830879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.766363] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6065.160767] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4044 to 0x2c0000400:4193) [ 6065.160791] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4044 to 0x280000400:4193) [ 6067.697328] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6068.480816] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6102.917234] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 19:56:24 (1784418984) [ 6107.006167] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6110.710588] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6111.643504] Lustre: Failing over lustre-MDT0000 [ 6111.922091] Lustre: server umount lustre-MDT0000 complete [ 6113.644681] LustreError: 151577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784418996 with bad export cookie 9436163628509976616 [ 6113.647078] Lustre: Failing over lustre-MDT0001 [ 6113.652671] LustreError: 151577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6113.783143] Lustre: server umount lustre-MDT0001 complete [ 6129.935185] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6129.938401] LDISKFS-fs (dm-0): recovery complete [ 6129.939424] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6129.943326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6129.944482] LDISKFS-fs (dm-1): recovery complete [ 6129.955128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.191212] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784418998/real 1784418998] req@ffff95b4ccc3d880 x1871092627071616/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784419014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6132.200141] Lustre: 3619:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 17 previous similar messages [ 6139.359607] LustreError: 3617:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95b4ccc3d880 x1871092627074176/t0(0) o250->MGC192.168.203.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6141.037384] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6141.113745] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6144.450481] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4044 to 0x2c0000400:4225) [ 6144.450513] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4044 to 0x280000400:4225) [ 6151.641683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10307 to 0x280000401:10753) [ 6151.641782] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10339 to 0x2c0000401:10721) [ 6154.441356] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6155.174262] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6155.833470] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6160.398589] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 19:57:22 (1784419042) [ 6161.256215] Lustre: 250559:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting cefe474c-7752-4514-aca7-3214b32ed13e at adminstrative request [ 6165.562803] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 19:57:27 (1784419047) [ 6168.568760] Lustre: Failing over lustre-MDT0000 [ 6168.851531] Lustre: server umount lustre-MDT0000 complete [ 6182.903544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.832867] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6188.591914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10804 to 0x280000401:10849) [ 6188.592082] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10773 to 0x2c0000401:10817) [ 6191.199613] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6191.897560] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6196.183631] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 19:57:58 (1784419078) [ 6201.338539] Lustre: Failing over lustre-OST0000 [ 6201.424784] Lustre: server umount lustre-OST0000 complete [ 6215.460316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6218.156307] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6222.999107] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6223.735622] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6228.678373] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 19:58:30 (1784419110) [ 6230.516585] Lustre: Failing over lustre-MDT0000 [ 6230.540846] LustreError: 151530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784419113 with bad export cookie 9436163628509989251 [ 6230.649455] Lustre: server umount lustre-MDT0000 complete [ 6234.126372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.101430] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6237.188314] Lustre: lustre-MDT0000: Denying connection for new client 43ff23b1-19d1-40fb-bbba-ee44984c274e (at 192.168.203.38@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 6237.198121] Lustre: Skipped 2 previous similar messages [ 6239.758386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10804 to 0x280000401:10881) [ 6239.758602] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10773 to 0x2c0000401:10849) [ 6245.420180] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 19:58:47 (1784419127) [ 6249.386825] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6250.493698] Lustre: Failing over lustre-OST0000 [ 6250.546412] Lustre: server umount lustre-OST0000 complete [ 6265.979652] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6265.980801] LDISKFS-fs (dm-2): recovery complete [ 6265.983710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6268.602497] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6273.456855] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6274.184144] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6278.496913] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 19:59:20 (1784419160) [ 6282.450478] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6284.721206] Lustre: Failing over lustre-OST0000 [ 6284.771831] Lustre: server umount lustre-OST0000 complete [ 6300.146391] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6300.149086] LDISKFS-fs (dm-2): recovery complete [ 6300.154668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6302.140328] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.38@tcp inode [0x2000301a1:0x5:0x0] object 0x280000401:10883 extent [0-4194303]: client csum f755cb12, server csum 7e06933e [ 6302.691982] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6306.796492] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6307.356369] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6311.084664] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 19:59:53 (1784419193) [ 6314.176920] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6317.439581] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6320.361970] Lustre: Failing over lustre-MDT0000 [ 6320.620720] Lustre: server umount lustre-MDT0000 complete [ 6322.318164] LustreError: 43767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784419204 with bad export cookie 9436163628510003482 [ 6322.321300] Lustre: Failing over lustre-OST0000 [ 6322.325985] LustreError: 43767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6322.399836] Lustre: server umount lustre-OST0000 complete [ 6338.807358] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6338.810079] LDISKFS-fs (dm-0): recovery complete [ 6338.815160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.232729] LustreError: 3617:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff95b6005a2680 x1871092627236864/t0(0) o250->MGC192.168.203.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6349.294255] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6352.589928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10773 to 0x2c0000401:10881) [ 6365.577561] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6365.579500] LDISKFS-fs (dm-2): recovery complete [ 6365.584280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6368.207679] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6375.246198] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 20:00:57 (1784419257) [ 6380.545622] Lustre: Failing over lustre-OST0000 [ 6380.622259] Lustre: server umount lustre-OST0000 complete [ 6382.415792] Lustre: Failing over lustre-MDT0000 [ 6382.501442] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.38@tcp (stopping) [ 6382.503204] Lustre: Skipped 7 previous similar messages [ 6382.640646] Lustre: server umount lustre-MDT0000 complete [ 6396.613892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6398.654084] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6402.058756] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10773 to 0x2c0000401:10913) [ 6405.990134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6408.748432] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6409.816798] Lustre: lustre-OST0000: Denying connection for new client 55e21beb-69a5-4b99-ac66-2eacdf9b8d9c (at 192.168.203.38@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 6415.269315] Lustre: lustre-OST0000: Denying connection for new client 55e21beb-69a5-4b99-ac66-2eacdf9b8d9c (at 192.168.203.38@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 6425.509241] Lustre: lustre-OST0000: Denying connection for new client 55e21beb-69a5-4b99-ac66-2eacdf9b8d9c (at 192.168.203.38@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:51 [ 6425.519380] Lustre: Skipped 1 previous similar message [ 6445.988867] Lustre: lustre-OST0000: Denying connection for new client 55e21beb-69a5-4b99-ac66-2eacdf9b8d9c (at 192.168.203.38@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:31 [ 6445.999038] Lustre: Skipped 3 previous similar messages [ 6477.500162] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6477.504190] Lustre: 264110:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 43ff23b1-19d1-40fb-bbba-ee44984c274e@ [ 6477.511716] Lustre: 264110:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6477.517029] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6477.525128] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6477.525183] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 6477.529425] Lustre: Skipped 54 previous similar messages [ 6477.534584] Lustre: Skipped 17 previous similar messages [ 6477.539774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10932 to 0x280000401:10953) [ 6483.655273] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 6490.542480] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 6493.974844] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 20:02:55 (1784419375) [ 6496.055654] Lustre: Failing over lustre-OST0001 [ 6496.128485] Lustre: server umount lustre-OST0001 complete [ 6497.203302] LustreError: 184023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6497.213024] LustreError: 184023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 209 previous similar messages [ 6498.273470] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6498.273515] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6498.278508] LustreError: Skipped 12 previous similar messages [ 6498.286892] Lustre: Skipped 53 previous similar messages [ 6510.730520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6510.803035] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6510.807093] Lustre: Skipped 18 previous similar messages [ 6510.811771] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [ 6510.814473] Lustre: Skipped 18 previous similar messages [ 6512.161619] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6512.165375] Lustre: Skipped 18 previous similar messages [ 6513.450551] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6519.098709] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 20:03:20 (1784419400) [ 6520.936599] Lustre: Failing over lustre-OST0000 [ 6520.994792] Lustre: server umount lustre-OST0000 complete [ 6535.080186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6537.006100] LustreError: 267213:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6537.011151] LustreError: 267213:0:(ldlm_lib.c:2904:target_recovery_thread()) Skipped 77 previous similar messages [ 6537.862814] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6542.815203] Lustre: *** cfs_fail_loc=715, val=40*** [ 6552.511380] Lustre: lustre-OST0000: Client 55e21beb-69a5-4b99-ac66-2eacdf9b8d9c (at 192.168.203.38@tcp) reconnected, waiting for 3 clients in recovery for 1:24 [ 6553.055161] Lustre: 3617:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784419419/real 1784419419] req@ffff95b5f4fcdc00 x1871092627352576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1784419435 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 6553.072737] Lustre: 3617:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 6553.078626] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [ 6553.085581] Lustre: Skipped 1 previous similar message [ 6558.687253] Lustre: *** cfs_fail_loc=715, val=40*** [ 6558.690214] Lustre: Skipped 2 previous similar messages [ 6568.415824] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:09 [ 6568.423314] Lustre: Skipped 1 previous similar message [ 6574.559157] Lustre: *** cfs_fail_loc=715, val=40*** [ 6574.561971] Lustre: Skipped 2 previous similar messages [ 6577.055115] LustreError: 267213:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6577.061770] LustreError: 267213:0:(ldlm_lib.c:2904:target_recovery_thread()) Skipped 77 previous similar messages [ 6579.678388] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6580.498680] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6585.120805] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 20:04:26 (1784419466) [ 6587.103313] Lustre: Failing over lustre-MDT0000 [ 6587.403901] Lustre: server umount lustre-MDT0000 complete [ 6600.639477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6600.704971] LustreError: MGC192.168.203.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6600.711699] LustreError: Skipped 9 previous similar messages [ 6602.576008] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6606.315296] LustreError: 268767:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6610.911280] Lustre: *** cfs_fail_loc=715, val=80*** [ 6610.914948] Lustre: Skipped 2 previous similar messages [ 6621.093083] Lustre: lustre-MDT0000: Client 55e21beb-69a5-4b99-ac66-2eacdf9b8d9c (at 192.168.203.38@tcp) reconnected, waiting for 2 clients in recovery for 1:25 [ 6621.100071] Lustre: Skipped 1 previous similar message [ 6621.664198] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6627.295161] Lustre: *** cfs_fail_loc=715, val=80*** [ 6627.297638] Lustre: Skipped 1 previous similar message [ 6636.453435] Lustre: lustre-MDT0000: Client 55e21beb-69a5-4b99-ac66-2eacdf9b8d9c (at 192.168.203.38@tcp) reconnected, waiting for 2 clients in recovery for 1:10 [ 6638.047674] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6642.655179] Lustre: *** cfs_fail_loc=715, val=80*** [ 6642.657747] Lustre: Skipped 1 previous similar message [ 6652.836856] Lustre: lustre-MDT0000: Client 55e21beb-69a5-4b99-ac66-2eacdf9b8d9c (at 192.168.203.38@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 6653.408104] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6659.039272] Lustre: *** cfs_fail_loc=715, val=80*** [ 6659.042348] Lustre: Skipped 1 previous similar message [ 6669.220797] Lustre: lustre-MDT0000: Client 55e21beb-69a5-4b99-ac66-2eacdf9b8d9c (at 192.168.203.38@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [ 6669.791648] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6686.175699] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6686.407122] LustreError: 268767:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6686.416186] Lustre: 268767:0:(ldlm_lib.c:2950:target_recovery_thread()) too long recovery - read logs [ 6686.422153] LustreError: dumping log to /tmp/lustre-log.1784419569.268767 [ 6686.521093] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10926 to 0x2c0000401:10945) [ 6686.521119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10967 to 0x280000401:10985) [ 6689.453260] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6690.305629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6695.425412] Lustre: DEBUG MARKER: == replay-single test complete, duration 6544 sec ======== 20:06:17 (1784419577) [ 6696.261777] Lustre: DEBUG MARKER: === replay-single: start cleanup 20:06:18 (1784419578) === [ 6700.490873] Lustre: DEBUG MARKER: === replay-single: finish cleanup 20:06:22 (1784419582) === [ 6736.730894] Lustre: server umount lustre-MDT0000 complete [ 6740.333577] LustreError: 151530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784419622 with bad export cookie 9436163628510016236 [ 6740.343788] LustreError: 151530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6740.490884] Lustre: server umount lustre-MDT0001 complete [ 6754.767045] Lustre: server umount lustre-OST0000 complete [ 6758.766818] Lustre: server umount lustre-OST0001 complete [ 6766.147883] Lustre: DEBUG MARKER: oleg338-server.virtnet: executing unload_modules_local [ 6767.279688] Key type lgssc unregistered [ 6767.414376] LNet: 272057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6767.418438] LNetError: 272057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6767.426619] LNet: Removed LNI 192.168.203.138@tcp [ 6767.816170] Key type .llcrypt unregistered [ 6767.817825] Key type ._llcrypt unregistered