[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476057755 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003224] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010110] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.014081] *** VALIDATE selinux *** [ 0.022814] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027072] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030101] *** VALIDATE tmpfs *** [ 0.032124] *** VALIDATE proc *** [ 0.033243] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036145] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037174] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.042638] debug: unmapping init [mem 0xffffffffb0459000-0xffffffffb0460fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047012] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051018] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053325] rcu: Hierarchical SRCU implementation. [ 0.055382] smp: Bringing up secondary CPUs ... [ 0.056605] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.060465] smp: Brought up 1 node, 4 CPUs [ 0.062014] smpboot: Max logical packages: 1 [ 0.063020] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.144397] node 0 deferred pages initialised in 77ms [ 0.148160] devtmpfs: initialized [ 0.149244] x86/mm: Memory block size: 128MB [ 0.151699] gcov: version magic: 0x41383552 [ 0.153110] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155266] pinctrl core: initialized pinctrl subsystem [ 0.156162] [ 0.156657] ************************************************************* [ 0.157014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158011] ** ** [ 0.159012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160011] ** ** [ 0.161012] ** This means that this kernel is built to expose internal ** [ 0.162011] ** IOMMU data structures, which may compromise security on ** [ 0.163012] ** your system. ** [ 0.164012] ** ** [ 0.165013] ** If you see this message and you are not debugging the ** [ 0.166015] ** kernel, report this immediately to your vendor! ** [ 0.167012] ** ** [ 0.168011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169015] ************************************************************* [ 0.170768] NET: Registered protocol family 16 [ 0.171453] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174353] cpuidle: using governor menu [ 0.177588] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180649] PCI: Using configuration type 1 for base access [ 0.183119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197167] cryptd: max_cpu_qlen set to 1000 [ 0.199233] ACPI: Added _OSI(Module Device) [ 0.200012] ACPI: Added _OSI(Processor Device) [ 0.201026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.203016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207539] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213028] ACPI: Interpreter enabled [ 0.214054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215008] ACPI: Using IOAPIC for interrupt routing [ 0.217230] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224305] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232310] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.239075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243653] acpiphp: Slot [2] registered [ 0.245136] acpiphp: Slot [5] registered [ 0.247121] acpiphp: Slot [6] registered [ 0.248131] acpiphp: Slot [7] registered [ 0.249131] acpiphp: Slot [8] registered [ 0.251102] acpiphp: Slot [9] registered [ 0.252111] acpiphp: Slot [10] registered [ 0.253113] acpiphp: Slot [3] registered [ 0.255120] acpiphp: Slot [4] registered [ 0.256133] acpiphp: Slot [11] registered [ 0.257085] acpiphp: Slot [12] registered [ 0.258196] acpiphp: Slot [13] registered [ 0.260118] acpiphp: Slot [14] registered [ 0.261094] acpiphp: Slot [15] registered [ 0.262000] acpiphp: Slot [16] registered [ 0.262000] acpiphp: Slot [17] registered [ 0.265132] acpiphp: Slot [18] registered [ 0.266102] acpiphp: Slot [19] registered [ 0.268165] acpiphp: Slot [20] registered [ 0.269113] acpiphp: Slot [21] registered [ 0.270124] acpiphp: Slot [22] registered [ 0.272117] acpiphp: Slot [23] registered [ 0.274162] acpiphp: Slot [24] registered [ 0.275153] acpiphp: Slot [25] registered [ 0.276101] acpiphp: Slot [26] registered [ 0.278156] acpiphp: Slot [27] registered [ 0.279113] acpiphp: Slot [28] registered [ 0.281178] acpiphp: Slot [29] registered [ 0.283121] acpiphp: Slot [30] registered [ 0.284101] acpiphp: Slot [31] registered [ 0.286099] PCI host bridge to bus 0000:00 [ 0.287022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.290023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.292023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.295019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.297017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.299021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.304093] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.307082] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.317016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.322012] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.324022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.326033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.329018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.332436] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.334811] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.337045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.341123] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.346012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.361019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.366920] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373568] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.381013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.388013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.415016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.424885] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.431015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.441016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.462015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.473584] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.486021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.495023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.523018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.533015] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.546017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.553018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.572018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.581000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.588013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.592014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.610016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.620898] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.633017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.640016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.661030] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.680504] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.681000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.683375] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.685353] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.688300] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.693148] iommu: Default domain type: Passthrough [ 0.695648] SCSI subsystem initialized [ 0.697210] ACPI: bus type USB registered [ 0.699153] usbcore: registered new interface driver usbfs [ 0.701247] usbcore: registered new interface driver hub [ 0.703097] usbcore: registered new device driver usb [ 0.705142] pps_core: LinuxPPS API ver. 1 registered [ 0.707008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.709078] PTP clock support registered [ 0.712105] EDAC MC: Ver: 3.0.0 [ 0.714164] PCI: Using ACPI for IRQ routing [ 0.717115] NetLabel: Initializing [ 0.718010] NetLabel: domain hash size = 128 [ 0.720009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.721073] NetLabel: unlabeled traffic allowed by default [ 0.723144] vgaarb: loaded [ 0.725285] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.727021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.733867] clocksource: Switched to clocksource kvm-clock [ 0.842948] VFS: Disk quotas dquot_6.6.0 [ 0.844474] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.847041] *** VALIDATE ramfs *** [ 0.848198] *** VALIDATE hugetlbfs *** [ 0.850068] pnp: PnP ACPI init [ 0.852314] pnp: PnP ACPI: found 6 devices [ 0.893222] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.896596] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.899153] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.901174] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.903717] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.906449] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.909350] NET: Registered protocol family 2 [ 0.912169] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.917154] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921296] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.927626] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.931633] TCP: Hash tables configured (established 65536 bind 65536) [ 0.935046] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.938948] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941867] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944895] NET: Registered protocol family 1 [ 0.948165] RPC: Registered named UNIX socket transport module. [ 0.951773] RPC: Registered udp transport module. [ 0.954540] RPC: Registered tcp transport module. [ 0.957253] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.961276] NET: Registered protocol family 44 [ 0.963747] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.967136] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.970227] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.973917] PCI: CLS 0 bytes, default 64 [ 0.976287] Unpacking initramfs... [ 2.405125] debug: unmapping init [mem 0xffff8cadfcc54000-0xffff8cadfffbffff] [ 2.410597] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.414413] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.419116] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 2.921939] Initialise system trusted keyrings [ 2.924125] Key type blacklist registered [ 2.926366] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.936033] zbud: loaded [ 2.939503] *** VALIDATE nfs *** [ 2.940860] *** VALIDATE nfs4 *** [ 2.942585] pstore: using deflate compression [ 2.946606] Platform Keyring initialized [ 3.055264] NET: Registered protocol family 38 [ 3.057104] Key type asymmetric registered [ 3.058884] Asymmetric key parser 'x509' registered [ 3.061158] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.065135] io scheduler mq-deadline registered [ 3.066935] io scheduler kyber registered [ 3.068442] io scheduler bfq registered [ 3.070287] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.073612] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.076530] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.079372] ACPI: Power Button [PWRF] [ 3.085227] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.091931] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.105737] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.112156] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.136096] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.166467] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.195368] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.201198] Non-volatile memory driver v1.3 [ 3.202850] Linux agpgart interface v0.103 [ 3.239833] virtio_blk virtio1: [vda] 134840 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.243923] vda: detected capacity change from 0 to 69038080 [ 3.257351] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.260213] vdb: detected capacity change from 0 to 1073741824 [ 3.274797] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.278640] vdc: detected capacity change from 0 to 2621440000 [ 3.293694] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.297016] vdd: detected capacity change from 0 to 2621440000 [ 3.311701] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.314986] vde: detected capacity change from 0 to 4294967296 [ 3.328894] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.333277] vdf: detected capacity change from 0 to 4294967296 [ 3.342559] libphy: Fixed MDIO Bus: probed [ 3.348666] usbcore: registered new interface driver usbserial_generic [ 3.351637] usbserial: USB Serial support registered for generic [ 3.354254] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.359227] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.361175] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.363964] mousedev: PS/2 mouse device common for all mice [ 3.367287] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.370928] rtc_cmos 00:05: RTC can wake from S4 [ 3.374240] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.376558] rtc_cmos 00:05: registered as rtc0 [ 3.380124] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.382977] intel_pstate: CPU model not supported [ 3.385401] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.388059] hid: raw HID events driver (C) Jiri Kosina [ 3.391086] usbcore: registered new interface driver usbhid [ 3.393081] usbhid: USB HID core driver [ 3.394859] drop_monitor: Initializing network drop monitor service [ 3.397265] Initializing XFRM netlink socket [ 3.399370] NET: Registered protocol family 10 [ 3.402181] Segment Routing with IPv6 [ 3.404350] NET: Registered protocol family 17 [ 3.407322] mpls_gso: MPLS GSO support [ 3.414846] RAS: Correctable Errors collector initialized. [ 3.418041] AVX version of gcm_enc/dec engaged. [ 3.420601] AES CTR mode by8 optimization enabled [ 3.510383] sched_clock: Marking stable (3510357724, 0)->(4470015137, -959657413) [ 3.513941] registered taskstats version 1 [ 3.515937] Loading compiled-in X.509 certificates [ 3.518226] zswap: loaded using pool lzo/zbud [ 3.548139] Key type big_key registered [ 3.561632] Key type encrypted registered [ 3.563613] ima: No TPM chip found, activating TPM-bypass! [ 3.565689] ima: Allocated hash algorithm: sha1 [ 3.567543] ima: No architecture policies found [ 3.569349] evm: Initialising EVM extended attributes: [ 3.571574] evm: security.selinux [ 3.572981] evm: security.ima [ 3.573959] evm: security.capability [ 3.575187] evm: HMAC attrs: 0x1 [ 3.577260] rtc_cmos 00:05: setting system clock to 2026-06-01 18:08:30 UTC (1780337310) [ 3.583263] debug: unmapping init [mem 0xffffffffb1403000-0xffffffffb15fffff] [ 3.586597] debug: unmapping init [mem 0xffffffffb0182000-0xffffffffb0458fff] [ 3.593386] Write protecting the kernel read-only data: 28672k [ 3.596434] debug: unmapping init [mem 0xffffffffae803000-0xffffffffae9fffff] [ 3.598263] debug: unmapping init [mem 0xffffffffaf114000-0xffffffffaf1fffff] [ 3.636308] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.648934] systemd[1]: Detected virtualization kvm. [ 3.651117] systemd[1]: Detected architecture x86-64. [ 3.653808] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.681760] systemd[1]: No hostname configured. [ 3.683739] systemd[1]: Set hostname to . [ 3.685971] random: systemd: uninitialized urandom read (16 bytes read) [ 3.688404] systemd[1]: Initializing machine ID from random generator. [ 3.828562] random: systemd: uninitialized urandom read (16 bytes read) [ 3.831733] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.837371] random: systemd: uninitialized urandom read (16 bytes read) [ 3.840434] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.845491] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.516301] device-mapper: uevent: version 1.0.3 [ 4.519284] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [[ 5.147788] random: fast init done  OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.307354] virtio_net virtio0 ens2: renamed from eth0 [ 5.381575] scsi host0: ata_piix [ 5.405362] scsi host1: ata_piix [ 5.407243] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.410258] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.189933] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.079324] random: crng init done [ 10.080701] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.538213] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.649163] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.930714] SELinux: Disabled at runtime. [ 11.989506] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.996418] systemd[1]: Detected virtualization kvm. [ 11.997778] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.479418] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.482872] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.487971] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.494597] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.497804] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.505130] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.512717] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ 12.688028] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.946248] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.285797] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.311837] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.367362] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.394671] EDAC sbridge: Ver: 1.1.2 [ 16.905303] Key type dns_resolver registered [ 17.731725] NFS: Registering the id_resolver key type [ 17.741704] Key type id_resolver registered [ 17.749899] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (5s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg438-server login: [ 75.827803] libcfs: loading out-of-tree module taints kernel. [ 75.859199] Key type ._llcrypt registered [ 75.868711] Key type .llcrypt registered [ 75.981116] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_hostid [ 98.177925] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 100.323797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 100.336757] alg: No test for adler32 (adler32-zlib) [ 101.994034] Lustre: Lustre: Build Version: 2.17.53_28_g0d90632 [ 102.901423] LNet: Added LNI 192.168.204.138@tcp [8/256/0/180] [ 104.655209] Key type lgssc registered [ 106.716140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 123.612083] hrtimer: interrupt took 2636415 ns [ 129.418820] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 175.489211] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing load_modules_local [ 190.395871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 190.460525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.849961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 191.877669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 191.996669] Lustre: lustre-MDT0000: new disk, initializing [ 192.101709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.119942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 196.293504] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.031707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.183664] Lustre: 6520:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 210.239727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 210.253531] Lustre: Skipped 1 previous similar message [ 210.406525] Lustre: lustre-MDT0001: new disk, initializing [ 210.493299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.517498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 210.528969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 215.044368] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.025330] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 229.602248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.945797] Lustre: lustre-OST0000: new disk, initializing [ 229.950547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 229.960382] Lustre: 8423:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 230.064228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.546560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.557336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 232.675233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 236.447862] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.959409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.099203] Lustre: lustre-OST0001: new disk, initializing [ 252.106511] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 252.114401] Lustre: 9480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.199771] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 258.479360] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.171765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.192281] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.246156] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 271.609218] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 277.751845] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 284.812085] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing check_logdir /tmp/testlogs/ [ 289.927657] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing yml_node [ 294.894475] Lustre: DEBUG MARKER: Client: 2.17.53.28 [ 297.691709] Lustre: DEBUG MARKER: MDS: 2.17.53.28 [ 300.847803] Lustre: DEBUG MARKER: OSS: 2.17.53.28 [ 302.656550] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Mon Jun 1 14:13:27 EDT 2026 [ 319.615217] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 321.102834] Lustre: DEBUG MARKER: === replay-single: start setup 14:13:46 (1780337626) === [ 325.486905] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing check_config_client /mnt/lustre [ 348.397672] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 352.121932] Lustre: 13193:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 355.985350] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 361.718184] Lustre: DEBUG MARKER: === replay-single: finish setup 14:14:26 (1780337666) === [ 364.453375] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 14:14:28 (1780337668) [ 372.364986] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 374.288705] Lustre: Failing over lustre-MDT0000 [ 374.489204] Lustre: server umount lustre-MDT0000 complete [ 378.848312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 378.862719] Lustre: Skipped 1 previous similar message [ 383.969239] LustreError: 6533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 383.989106] LustreError: 6533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 389.092201] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.110096] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 394.209351] LustreError: 14065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 394.210803] Lustre: 3669:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337685/real 1780337685] req@ffff8cad42c93480 x1866819079098624/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780337701 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 394.238373] LustreError: 14065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 394.283479] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 396.348485] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 396.353933] LDISKFS-fs (dm-0): recovery complete [ 396.373437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.329545] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.359944] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 404.449223] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae7d9f6d80 x1866819079106176/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 404.485962] LustreError: 6528:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 404.507990] LustreError: 6528:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 404.785530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.825362] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 409.675912] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 410.110441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 410.179180] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 418.218354] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 420.130833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 428.727099] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 14:15:33 (1780337733) [ 431.274730] Lustre: Failing over lustre-OST0000 [ 431.433460] LustreError: 8406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.493803] Lustre: server umount lustre-OST0000 complete [ 432.618167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 432.624973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 432.638481] Lustre: Skipped 2 previous similar messages [ 435.689771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.096694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 450.307929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.619780] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 451.913627] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 451.914526] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 451.936385] Lustre: Skipped 3 previous similar messages [ 456.780055] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 464.732743] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 466.471517] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 476.598094] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 14:16:21 (1780337781) [ 484.567698] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 486.892304] Lustre: Failing over lustre-MDT0000 [ 487.119143] Lustre: server umount lustre-MDT0000 complete [ 491.493668] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.494660] LustreError: 6532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.506488] Lustre: Skipped 2 previous similar messages [ 491.517034] LustreError: 6532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 507.350198] Lustre: 3670:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337798/real 1780337798] req@ffff8cad4409d880 x1866819079161600/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780337814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 507.388972] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.067475] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 510.070897] LDISKFS-fs (dm-0): recovery complete [ 510.085851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.600902] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae6f55b480 x1866819079171072/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 517.631743] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35bcca2 [ 517.644837] Lustre: MGC192.168.204.138@tcp: Connection restored to 0@lo (at 0@lo) [ 517.656284] Lustre: Skipped 1 previous similar message [ 517.835797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.511680] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 523.245335] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 523.253233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 524.742863] Lustre: lustre-MDT0000: Denying connection for new client f154a618-1e9b-41f1-8a40-9dd5c52a6b46 (at 192.168.204.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 530.221209] Lustre: lustre-MDT0000: Denying connection for new client f154a618-1e9b-41f1-8a40-9dd5c52a6b46 (at 192.168.204.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 535.344179] Lustre: lustre-MDT0000: Denying connection for new client f154a618-1e9b-41f1-8a40-9dd5c52a6b46 (at 192.168.204.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 540.463970] Lustre: lustre-MDT0000: Denying connection for new client f154a618-1e9b-41f1-8a40-9dd5c52a6b46 (at 192.168.204.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 545.579206] Lustre: lustre-MDT0000: Denying connection for new client f154a618-1e9b-41f1-8a40-9dd5c52a6b46 (at 192.168.204.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:47 [ 555.819276] Lustre: lustre-MDT0000: Denying connection for new client f154a618-1e9b-41f1-8a40-9dd5c52a6b46 (at 192.168.204.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:37 [ 555.838545] Lustre: Skipped 1 previous similar message [ 576.299731] Lustre: lustre-MDT0000: Denying connection for new client f154a618-1e9b-41f1-8a40-9dd5c52a6b46 (at 192.168.204.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:17 [ 576.316556] Lustre: Skipped 3 previous similar messages [ 593.500806] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 593.507390] Lustre: 17855:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a35045da-5c5d-4285-9f81-367ff2779bfa@ [ 593.537954] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 593.553632] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 593.555638] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 593.569095] Lustre: Skipped 2 previous similar messages [ 593.603419] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 593.603802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 605.957660] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 14:18:31 (1780337911) [ 613.821424] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 616.104941] Lustre: Failing over lustre-MDT0000 [ 616.592281] Lustre: server umount lustre-MDT0000 complete [ 619.491187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 619.500395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.512166] Lustre: Skipped 1 previous similar message [ 619.523109] LustreError: 6533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.541228] LustreError: 6533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 636.865201] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337927/real 1780337927] req@ffff8cae7c894e00 x1866819079222656/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780337943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 636.904959] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.438924] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 638.446270] LDISKFS-fs (dm-0): recovery complete [ 638.456615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.114439] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae7cac5180 x1866819079230464/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 646.432958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.475841] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 651.348691] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 651.753289] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 651.761242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 653.897642] Lustre: lustre-MDT0000: Denying connection for new client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 653.919369] Lustre: Skipped 3 previous similar messages [ 720.680745] Lustre: lustre-MDT0000: Denying connection for new client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:00 [ 720.690222] Lustre: Skipped 12 previous similar messages [ 721.500213] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 721.509674] Lustre: 19564:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f154a618-1e9b-41f1-8a40-9dd5c52a6b46@ [ 721.521259] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 721.529522] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 721.529665] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 721.543055] Lustre: Skipped 2 previous similar messages [ 721.580737] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 721.581622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 734.513365] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 14:20:39 (1780338039) [ 742.158226] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 744.209643] Lustre: Failing over lustre-MDT0000 [ 744.540325] Lustre: server umount lustre-MDT0000 complete [ 746.320351] LustreError: 14697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 746.355144] LustreError: 14697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 28 previous similar messages [ 747.490826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 747.500070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 747.537105] Lustre: Skipped 3 previous similar messages [ 764.389255] Lustre: 3670:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780338055/real 1780338055] req@ffff8cae7cac4e00 x1866819079280256/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780338071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 764.409986] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 766.848582] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 766.853228] LDISKFS-fs (dm-0): recovery complete [ 766.860479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 774.885609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 774.943149] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 776.465173] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 779.532263] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 780.283399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 780.340966] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 780.390137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 780.392355] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 788.750224] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 790.891841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 800.933310] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 14:21:45 (1780338105) [ 809.262479] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 811.356378] Lustre: Failing over lustre-MDT0000 [ 811.639090] Lustre: server umount lustre-MDT0000 complete [ 816.098588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.100462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 816.113335] Lustre: Skipped 5 previous similar messages [ 832.479193] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780338123/real 1780338123] req@ffff8cad42d40700 x1866819079315584/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780338139 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 832.498441] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 834.556619] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 834.559972] LDISKFS-fs (dm-0): recovery complete [ 834.571072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 842.723474] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae46df5500 x1866819079324032/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 843.028209] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 843.563234] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 847.726382] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 848.360941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 848.367922] Lustre: Skipped 3 previous similar messages [ 848.475362] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 848.526321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 848.533138] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 856.580798] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 858.521217] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 868.617125] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 14:22:53 (1780338173) [ 877.000387] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 879.102525] Lustre: Failing over lustre-MDT0000 [ 879.351988] Lustre: server umount lustre-MDT0000 complete [ 879.420226] LustreError: 17314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.437279] LustreError: 17314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 65 previous similar messages [ 879.591759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 879.600261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.321124] Lustre: 3668:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780338191/real 1780338191] req@ffff8cad442bed80 x1866819079353600/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780338207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 900.359839] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 901.996173] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 901.998542] LDISKFS-fs (dm-0): recovery complete [ 902.008724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.819511] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35bdfe9 [ 911.084952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 911.096792] Lustre: Skipped 1 previous similar message [ 911.134345] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 912.122897] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 915.533905] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 916.605880] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 916.668951] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:193) [ 916.671678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 923.550845] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 926.062031] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 935.251230] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 14:24:00 (1780338240) [ 942.720508] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 944.970951] Lustre: Failing over lustre-MDT0000 [ 945.193687] Lustre: server umount lustre-MDT0000 complete [ 947.169112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.182694] Lustre: Skipped 5 previous similar messages [ 963.943310] Lustre: 3668:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780338254/real 1780338254] req@ffff8cae46bcb100 x1866819079390464/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780338270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 963.989347] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 968.846846] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 968.854413] LDISKFS-fs (dm-0): recovery complete [ 968.869023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.569970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 973.617021] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 977.735042] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 978.931554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 978.957175] Lustre: Skipped 8 previous similar messages [ 979.112912] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 979.173330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 979.175853] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 985.974959] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 987.538536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 995.836318] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 14:25:00 (1780338300) [ 1002.974308] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1004.780394] Lustre: Failing over lustre-MDT0000 [ 1004.988592] Lustre: server umount lustre-MDT0000 complete [ 1005.029289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1026.022879] Lustre: 3668:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780338316/real 1780338316] req@ffff8cae45b98700 x1866819079426688/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780338332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1026.068095] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.087163] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1030.089464] LDISKFS-fs (dm-0): recovery complete [ 1030.109881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.256722] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae46bc9f80 x1866819079435776/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1036.429630] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1040.014717] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1041.977225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1041.982203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1048.011086] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1049.634830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1059.281463] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 14:26:03 (1780338363) [ 1060.199715] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1060.207318] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1060.220857] LustreError: 6530:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cad42cb4e00 x1866819056062336/t38654705666(0) o35->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:58/0 lens 392/456 e 0 to 0 dl 1780338378 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1069.584364] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1072.021405] Lustre: Failing over lustre-MDT0000 [ 1072.284039] Lustre: server umount lustre-MDT0000 complete [ 1072.623952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1093.089019] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780338384/real 1780338384] req@ffff8cae7d821f80 x1866819079464960/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780338400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1093.102649] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1095.578871] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1095.580686] LDISKFS-fs (dm-0): recovery complete [ 1095.587297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.329684] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae5cdf9880 x1866819079473024/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1103.743096] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1104.972875] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1104.984021] Lustre: Skipped 1 previous similar message [ 1108.095091] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1109.065069] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1109.083095] Lustre: Skipped 1 previous similar message [ 1109.104659] Lustre: 6530:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cad4132ed80 x1866819056062336/t38654705666(0) o35->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:107/0 lens 392/456 e 0 to 0 dl 1780338427 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1109.141749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1109.142919] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1116.837589] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1119.047989] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1127.872962] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 14:27:12 (1780338432) [ 1135.761614] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1138.089380] Lustre: Failing over lustre-MDT0000 [ 1138.597868] Lustre: server umount lustre-MDT0000 complete [ 1139.687565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1139.689659] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.698640] LustreError: 6532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.698651] LustreError: 6532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 139 previous similar messages [ 1139.747740] Lustre: Skipped 11 previous similar messages [ 1162.312152] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1162.314204] LDISKFS-fs (dm-0): recovery complete [ 1162.324879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.802435] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35bf3f4 [ 1166.184232] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1171.098643] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1171.582514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1171.585906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1179.171415] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1180.668448] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1189.268502] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 14:28:13 (1780338493) [ 1196.665200] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1197.758967] Lustre: *** cfs_fail_loc=114, val=0*** [ 1200.898412] Lustre: Failing over lustre-MDT0000 [ 1201.516966] Lustre: server umount lustre-MDT0000 complete [ 1225.248944] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1225.253278] LDISKFS-fs (dm-0): recovery complete [ 1225.267440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1228.772474] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35bf90a [ 1229.037342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1229.047153] Lustre: Skipped 4 previous similar messages [ 1233.963486] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1234.493506] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1234.494496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1242.296428] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1244.213871] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1254.166401] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 14:29:18 (1780338558) [ 1264.049640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1265.479684] Lustre: *** cfs_fail_loc=128, val=0*** [ 1268.267256] Lustre: Failing over lustre-MDT0000 [ 1268.491595] Lustre: server umount lustre-MDT0000 complete [ 1270.245159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1286.561826] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780338577/real 1780338577] req@ffff8cae7d9f7480 x1866819079574144/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780338593 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1286.597180] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1286.608487] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1286.622192] LustreError: Skipped 2 previous similar messages [ 1291.622157] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1291.625379] LDISKFS-fs (dm-0): recovery complete [ 1291.639097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1296.869684] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35bfdef [ 1296.896781] Lustre: MGC192.168.204.138@tcp: Connection restored to 0@lo (at 0@lo) [ 1296.917423] Lustre: Skipped 21 previous similar messages [ 1297.233143] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1297.246800] Lustre: Skipped 1 previous similar message [ 1298.647948] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1298.655344] Lustre: Skipped 2 previous similar messages [ 1301.023223] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1302.572735] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1302.583139] Lustre: Skipped 2 previous similar messages [ 1302.617475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1302.619857] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1309.527422] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1311.700727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1323.234727] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 14:30:27 (1780338627) [ 1331.512610] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1333.858066] Lustre: Failing over lustre-MDT0000 [ 1334.120770] Lustre: server umount lustre-MDT0000 complete [ 1356.806139] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1356.811946] LDISKFS-fs (dm-0): recovery complete [ 1356.826922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.455797] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae5fbc5c00 x1866819079622144/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1369.583749] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1370.652838] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1370.653686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1378.626602] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1380.799397] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1391.537523] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 14:31:36 (1780338696) [ 1400.748711] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1402.824527] Lustre: Failing over lustre-MDT0000 [ 1403.086392] Lustre: server umount lustre-MDT0000 complete [ 1405.921659] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.938073] Lustre: Skipped 15 previous similar messages [ 1426.487564] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1426.489636] LDISKFS-fs (dm-0): recovery complete [ 1426.500234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1432.896889] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1432.903465] Lustre: Skipped 1 previous similar message [ 1437.371912] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1438.422211] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1438.424519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1445.792221] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1447.785845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1457.215910] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 14:32:41 (1780338761) [ 1464.901725] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1474.589688] Lustre: Failing over lustre-MDT0000 [ 1476.877797] Lustre: server umount lustre-MDT0000 complete [ 1501.870142] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1501.872267] LDISKFS-fs (dm-0): recovery complete [ 1501.879892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.769183] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35c45a1 [ 1512.024880] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1515.376514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1515.384605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1520.475282] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1522.982280] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1551.146909] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 14:34:15 (1780338855) [ 1559.280587] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1561.516278] Lustre: Failing over lustre-MDT0000 [ 1561.571574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1561.576069] LustreError: Skipped 1 previous similar message [ 1561.587923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1561.872394] Lustre: server umount lustre-MDT0000 complete [ 1578.448424] Lustre: 3668:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780338869/real 1780338869] req@ffff8cae46bc9880 x1866819079798912/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780338885 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1578.496562] Lustre: 3668:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1578.506853] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1578.535572] LustreError: Skipped 3 previous similar messages [ 1585.700398] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1585.705577] LDISKFS-fs (dm-0): recovery complete [ 1585.724159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.568949] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1590.577820] Lustre: Skipped 3 previous similar messages [ 1594.227919] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1594.428327] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1594.434870] Lustre: Skipped 3 previous similar messages [ 1594.501229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1594.501591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1602.448611] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1604.301351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1616.781714] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 14:35:21 (1780338921) [ 1625.771127] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1627.647353] Lustre: Failing over lustre-MDT0000 [ 1627.847494] Lustre: server umount lustre-MDT0000 complete [ 1650.923832] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1650.929300] LDISKFS-fs (dm-0): recovery complete [ 1650.944160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.012664] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.026770] LustreError: 6529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 263 previous similar messages [ 1656.807566] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae7d847800 x1866819079850240/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1662.322570] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1662.577185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1662.579309] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1670.928364] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1672.867676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1682.981537] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 14:36:27 (1780338987) [ 1690.850816] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1692.939386] Lustre: Failing over lustre-MDT0000 [ 1692.986324] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.38@tcp (stopping) [ 1693.403898] Lustre: server umount lustre-MDT0000 complete [ 1716.297942] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1716.302046] LDISKFS-fs (dm-0): recovery complete [ 1716.313503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.779440] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35cb55b [ 1720.069231] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1720.081690] Lustre: Skipped 3 previous similar messages [ 1724.588647] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1725.510715] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1725.511032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1734.475659] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1737.018932] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1747.438691] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 14:37:32 (1780339052) [ 1757.569484] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1759.806038] Lustre: Failing over lustre-MDT0000 [ 1760.166139] Lustre: server umount lustre-MDT0000 complete [ 1783.716244] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1783.719201] LDISKFS-fs (dm-0): recovery complete [ 1783.731440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1788.386917] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35cbaf6 [ 1788.560895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.569303] Lustre: Skipped 7 previous similar messages [ 1793.322931] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1794.218978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1794.224186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1803.261785] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1805.951780] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1816.711257] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 14:38:41 (1780339121) [ 1825.529805] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1827.461572] Lustre: Failing over lustre-MDT0000 [ 1827.701537] Lustre: server umount lustre-MDT0000 complete [ 1850.506313] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1850.511904] LDISKFS-fs (dm-0): recovery complete [ 1850.526876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.662684] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1862.127680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1862.135260] Lustre: Skipped 35 previous similar messages [ 1862.250609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1862.251134] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1869.860485] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1871.793240] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1882.130629] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 14:39:46 (1780339186) [ 1890.553926] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1892.766417] Lustre: Failing over lustre-MDT0000 [ 1892.834843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1892.838451] Lustre: Skipped 2 previous similar messages [ 1892.842272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1892.848797] LustreError: Skipped 2 previous similar messages [ 1893.118539] Lustre: server umount lustre-MDT0000 complete [ 1916.800460] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1916.804155] LDISKFS-fs (dm-0): recovery complete [ 1916.810035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.557039] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35cc5b5 [ 1929.278655] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1929.281232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1929.373306] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1938.652939] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1940.910487] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1951.366171] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 14:40:56 (1780339256) [ 1960.584306] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1963.057324] Lustre: Failing over lustre-MDT0000 [ 1963.383463] Lustre: server umount lustre-MDT0000 complete [ 1965.028835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.048502] Lustre: Skipped 32 previous similar messages [ 1987.877732] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1987.880303] LDISKFS-fs (dm-0): recovery complete [ 1987.886408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1990.633497] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae5cff5c00 x1866819080040192/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1990.663625] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 1995.083373] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1996.488206] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1996.493546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 2003.748040] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2005.845842] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2015.520724] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 14:42:00 (1780339320) [ 2025.310500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2027.762327] Lustre: Failing over lustre-MDT0000 [ 2027.833448] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.38@tcp (stopping) [ 2027.836691] Lustre: Skipped 1 previous similar message [ 2028.159783] Lustre: server umount lustre-MDT0000 complete [ 2052.867431] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2052.869416] LDISKFS-fs (dm-0): recovery complete [ 2052.884662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.735325] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35cd146 [ 2063.787515] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2064.543971] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 2064.547454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 2072.803790] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2074.942245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2083.893932] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 14:43:08 (1780339388) [ 2093.760554] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2095.914937] Lustre: Failing over lustre-MDT0000 [ 2096.162591] Lustre: server umount lustre-MDT0000 complete [ 2117.094122] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780339407/real 1780339407] req@ffff8cae7d9f7480 x1866819080107392/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780339423 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2117.132379] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2117.144033] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2117.157896] LustreError: Skipped 7 previous similar messages [ 2120.506625] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2120.510775] LDISKFS-fs (dm-0): recovery complete [ 2120.520759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2127.327749] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae72a85f80 x1866819080115968/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2127.998232] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2128.004049] Lustre: Skipped 7 previous similar messages [ 2132.681199] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2133.027747] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 2133.039503] Lustre: Skipped 7 previous similar messages [ 2133.080720] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 2133.082638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 2141.356578] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2143.113841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2152.835727] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 14:44:17 (1780339457) [ 2161.244465] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2163.422645] Lustre: Failing over lustre-MDT0000 [ 2163.819410] Lustre: server umount lustre-MDT0000 complete [ 2188.540497] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2188.550331] LDISKFS-fs (dm-0): recovery complete [ 2188.560991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.945557] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2201.256846] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 2201.261158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:897) [ 2210.212328] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2212.721864] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2226.454972] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 14:45:29 (1780339529) [ 2237.605540] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2239.888241] Lustre: Failing over lustre-MDT0000 [ 2240.271385] Lustre: server umount lustre-MDT0000 complete [ 2252.256362] LustreError: 14697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.276194] LustreError: 14697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 291 previous similar messages [ 2264.760433] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2264.767530] LDISKFS-fs (dm-0): recovery complete [ 2264.775784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2267.981915] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2267.997150] Lustre: Skipped 7 previous similar messages [ 2272.828759] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2273.431386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2273.435658] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2281.111858] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2283.232405] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2292.430438] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 14:46:37 (1780339597) [ 2301.526427] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2304.112689] Lustre: Failing over lustre-MDT0000 [ 2304.429394] Lustre: server umount lustre-MDT0000 complete [ 2328.960760] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2328.962903] LDISKFS-fs (dm-0): recovery complete [ 2328.972767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.959559] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2341.035459] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 2341.039330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 2348.081210] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2350.196635] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2360.507935] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 14:47:45 (1780339665) [ 2368.950406] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2371.188330] Lustre: Failing over lustre-MDT0000 [ 2371.386486] Lustre: server umount lustre-MDT0000 complete [ 2396.565544] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2396.571271] LDISKFS-fs (dm-0): recovery complete [ 2396.583925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.046635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.053806] Lustre: Skipped 8 previous similar messages [ 2407.099914] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2408.503135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 2408.503471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:993) [ 2414.979661] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2416.573699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2425.463857] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 14:48:50 (1780339730) [ 2434.081995] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2436.533951] Lustre: Failing over lustre-MDT0000 [ 2436.684190] Lustre: server umount lustre-MDT0000 complete [ 2439.161804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2439.176386] LustreError: Skipped 3 previous similar messages [ 2461.991638] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2461.994690] LDISKFS-fs (dm-0): recovery complete [ 2462.005611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2465.252695] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae5f881880 x1866819080312064/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2465.279519] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 48 previous similar messages [ 2470.129268] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2470.908951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2470.918354] Lustre: Skipped 37 previous similar messages [ 2471.064430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2471.066218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2478.966610] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2480.941182] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2490.686819] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 14:49:55 (1780339795) [ 2497.735774] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2499.899078] Lustre: Failing over lustre-MDT0000 [ 2502.082343] Lustre: server umount lustre-MDT0000 complete [ 2523.813144] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2523.817392] LDISKFS-fs (dm-0): recovery complete [ 2523.827638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.279767] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2532.938839] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2532.943294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2538.717929] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2540.756825] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2550.763205] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 14:50:55 (1780339855) [ 2558.718675] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2561.142447] Lustre: Failing over lustre-MDT0000 [ 2561.372333] Lustre: server umount lustre-MDT0000 complete [ 2587.017515] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2587.020428] LDISKFS-fs (dm-0): recovery complete [ 2587.027657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.151805] LustreError: 70859:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2589.168415] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35cfe64 [ 2594.157346] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2595.062638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2595.062910] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2602.976737] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2604.740400] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2616.730184] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 14:52:01 (1780339921) [ 2622.269347] Lustre: 71767:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting bee07ca0-5f3b-4b44-b416-9964de954a49 at adminstrative request [ 2629.914278] Lustre: Failing over lustre-MDT0000 [ 2630.192966] Lustre: server umount lustre-MDT0000 complete [ 2630.624183] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2630.634684] Lustre: Skipped 36 previous similar messages [ 2650.365346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.957203] LustreError: 40998:0:(ofd_io.c:784:ofd_preprw_write()) lustre-OST0000: BRW to missing obj 0x280000401:1090 [ 2661.952957] LustreError: 8413:0:(ofd_io.c:784:ofd_preprw_write()) lustre-OST0000: BRW to missing obj 0x280000401:1090 [ 2661.986040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2661.991185] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2662.119854] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2670.508611] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2672.673703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2677.949629] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2687.527980] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2695.921918] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 14:53:20 (1780340000) [ 2697.168952] Lustre: 73665:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting bee07ca0-5f3b-4b44-b416-9964de954a49 at adminstrative request [ 2706.922654] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 14:53:31 (1780340011) [ 2715.605827] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2717.841844] Lustre: Failing over lustre-MDT0000 [ 2718.124780] Lustre: server umount lustre-MDT0000 complete [ 2734.559274] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780340025/real 1780340025] req@ffff8cae45a45c00 x1866819080463104/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780340041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2734.576354] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2734.588813] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2734.602876] LustreError: Skipped 8 previous similar messages [ 2741.486649] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2741.494281] LDISKFS-fs (dm-0): recovery complete [ 2741.504178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.584648] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2746.596443] Lustre: Skipped 8 previous similar messages [ 2750.592035] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2750.601568] Lustre: Skipped 8 previous similar messages [ 2750.696328] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2750.697514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2750.742993] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2759.438506] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2761.853616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2773.155907] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 14:54:37 (1780340077) [ 2781.814337] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2783.759340] Lustre: Failing over lustre-MDT0000 [ 2784.235358] Lustre: server umount lustre-MDT0000 complete [ 2807.851632] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2807.856356] LDISKFS-fs (dm-0): recovery complete [ 2807.871431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2817.714354] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2818.183800] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2818.185038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2826.734380] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2829.289382] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2840.624819] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 14:55:45 (1780340145) [ 2850.498268] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2852.770166] Lustre: Failing over lustre-MDT0000 [ 2853.090510] Lustre: server umount lustre-MDT0000 complete [ 2853.860612] LustreError: 17314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.872135] LustreError: 17314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 293 previous similar messages [ 2876.123600] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2876.125599] LDISKFS-fs (dm-0): recovery complete [ 2876.140578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.496500] Lustre: MGS: Client c34d4833-0bbf-412b-8fd8-a9ee39049603 (at 0@lo) reconnecting [ 2880.877082] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2880.891349] Lustre: Skipped 8 previous similar messages [ 2886.261630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2886.261650] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2886.287633] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2895.346333] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2897.871397] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2910.588264] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 14:56:54 (1780340214) [ 2919.055399] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2921.368207] Lustre: Failing over lustre-MDT0000 [ 2921.797888] Lustre: server umount lustre-MDT0000 complete [ 2945.837753] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2945.839983] LDISKFS-fs (dm-0): recovery complete [ 2945.844951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.425249] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2958.436307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2958.852565] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2967.988787] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2970.211971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2981.046070] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 14:58:05 (1780340285) [ 2989.799222] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2992.255492] Lustre: Failing over lustre-MDT0000 [ 2992.542182] Lustre: server umount lustre-MDT0000 complete [ 3021.239397] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3021.241794] LDISKFS-fs (dm-0): recovery complete [ 3021.307196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.121529] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae5fb49880 x1866819080637824/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3035.145911] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 46 previous similar messages [ 3035.784690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3035.794945] Lustre: Skipped 8 previous similar messages [ 3040.850832] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 3040.854345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 3041.852098] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3050.770360] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3052.717924] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3062.590778] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 14:59:27 (1780340367) [ 3072.253562] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3074.885590] Lustre: Failing over lustre-MDT0000 [ 3075.458564] Lustre: server umount lustre-MDT0000 complete [ 3100.090140] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3100.092573] LDISKFS-fs (dm-0): recovery complete [ 3100.098909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3107.839175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3107.846906] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3107.851707] Lustre: Skipped 41 previous similar messages [ 3108.111777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 3108.112658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 3116.723847] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3118.684456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3127.672668] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 15:00:32 (1780340432) [ 3136.030930] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3138.444676] Lustre: Failing over lustre-MDT0000 [ 3138.545765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3138.552710] Lustre: Skipped 3 previous similar messages [ 3138.743875] Lustre: server umount lustre-MDT0000 complete [ 3162.716314] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3162.718366] LDISKFS-fs (dm-0): recovery complete [ 3162.747402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.177417] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3176.616774] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 3176.618202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 3184.341558] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3187.039154] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3198.448657] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 15:01:42 (1780340502) [ 3208.743461] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3210.565493] Lustre: Failing over lustre-MDT0000 [ 3210.836499] Lustre: server umount lustre-MDT0000 complete [ 3235.238463] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3235.242512] LDISKFS-fs (dm-0): recovery complete [ 3235.258480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.255608] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3244.623609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 3244.623609] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 3252.939492] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3255.109812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3265.448833] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 15:02:50 (1780340570) [ 3274.503431] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3276.969208] Lustre: Failing over lustre-MDT0000 [ 3277.354288] Lustre: server umount lustre-MDT0000 complete [ 3280.352261] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.369880] Lustre: Skipped 37 previous similar messages [ 3303.094381] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3303.101210] LDISKFS-fs (dm-0): recovery complete [ 3303.125870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3312.082353] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3312.778680] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3312.781143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3320.688553] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3322.610969] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3332.568313] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 15:03:57 (1780340637) [ 3341.169902] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3343.439726] Lustre: Failing over lustre-MDT0000 [ 3343.714749] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.38@tcp (stopping) [ 3343.840690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3343.847037] LustreError: Skipped 3 previous similar messages [ 3343.958478] Lustre: server umount lustre-MDT0000 complete [ 3364.833893] Lustre: 3670:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780340655/real 1780340655] req@ffff8cae6f6b8a80 x1866819080828672/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780340671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3364.858411] Lustre: 3670:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 3364.868263] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3364.881165] LustreError: Skipped 9 previous similar messages [ 3368.824176] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3368.827289] LDISKFS-fs (dm-0): recovery complete [ 3368.839394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.091405] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f35d3ce6 [ 3375.101919] Lustre: Skipped 4 previous similar messages [ 3376.832250] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3376.839872] Lustre: Skipped 8 previous similar messages [ 3379.579358] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3380.751228] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3380.759937] Lustre: Skipped 8 previous similar messages [ 3380.800431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3380.801793] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 3387.854813] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3390.787202] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3401.249539] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 15:05:05 (1780340705) [ 3410.258670] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3412.737731] Lustre: Failing over lustre-MDT0000 [ 3413.128417] Lustre: server umount lustre-MDT0000 complete [ 3436.179503] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3436.181821] LDISKFS-fs (dm-0): recovery complete [ 3436.189810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3447.267282] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3447.879643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 3447.880212] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 3457.206920] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3459.199706] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3469.914608] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 15:06:14 (1780340774) [ 3471.103940] Lustre: 94025:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting bee07ca0-5f3b-4b44-b416-9964de954a49 at adminstrative request [ 3482.206941] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 15:06:26 (1780340786) [ 3489.585921] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3491.619901] Lustre: Failing over lustre-MDT0000 [ 3491.977400] Lustre: server umount lustre-MDT0000 complete [ 3493.861198] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3493.877251] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 323 previous similar messages [ 3505.236631] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3505.238790] LDISKFS-fs (dm-0): recovery complete [ 3505.250595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3505.460248] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.38@tcp (not set up) [ 3505.759686] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3505.763190] Lustre: lustre-MDT0000: Aborting client recovery [ 3505.767673] Lustre: Skipped 8 previous similar messages [ 3505.769636] LustreError: 95307:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3505.772813] LustreError: 95339:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0000-osd: get update log duration 0, retries 0, failed: rc = -108 [ 3505.797280] Lustre: 95341:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3505.810138] Lustre: 95341:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client bee07ca0-5f3b-4b44-b416-9964de954a49@ [ 3505.826713] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3505.851173] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3505.860731] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3505.926715] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1505) [ 3505.929515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505) [ 3510.414752] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3510.767090] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3533.016816] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 15:07:17 (1780340837) [ 3542.353917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3544.584418] Lustre: Failing over lustre-MDT0000 [ 3545.030629] Lustre: server umount lustre-MDT0000 complete [ 3561.443805] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3561.446743] LDISKFS-fs (dm-0): recovery complete [ 3561.455048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.143908] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3562.188115] Lustre: lustre-MDT0000: Aborting client recovery [ 3562.192120] LustreError: 97157:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3562.212988] Lustre: 97191:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3562.232381] Lustre: 97191:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 3562.249342] Lustre: 97191:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3562.268515] Lustre: 97191:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3562.280241] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3562.299917] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3562.335364] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3562.412988] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1516 to 0x2c0000401:1537) [ 3562.419617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1516 to 0x280000401:1537) [ 3567.266387] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3567.588611] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3583.526319] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3583.528320] Lustre: Skipped 1 previous similar message [ 3594.010478] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 15:08:18 (1780340898) [ 3603.058777] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3605.648759] Lustre: Failing over lustre-MDT0000 [ 3605.993706] Lustre: server umount lustre-MDT0000 complete [ 3620.607286] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3620.613703] LDISKFS-fs (dm-0): recovery complete [ 3620.634730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.180227] Lustre: lustre-MDT0000: Aborting client recovery [ 3621.184128] LustreError: 99010:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3621.195637] Lustre: 99042:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3621.208619] Lustre: 99042:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 3621.228558] Lustre: 99042:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3621.245666] Lustre: 99042:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3621.250382] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3621.260557] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3621.278285] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3621.340337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 3621.341987] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 3626.476088] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3627.114874] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3651.962574] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 15:09:16 (1780340956) [ 3653.432401] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3653.437286] LustreError: 82961:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae7cacb480 x1866819057253504/t201863462916(0) o36->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:386/0 lens 512/456 e 0 to 0 dl 1780340971 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3658.367114] Lustre: Failing over lustre-MDT0000 [ 3659.045749] Lustre: server umount lustre-MDT0000 complete [ 3670.180253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.441266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3670.454081] Lustre: Skipped 9 previous similar messages [ 3670.480437] Lustre: lustre-MDT0000: Aborting client recovery [ 3670.483587] LustreError: 100320:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3670.488860] Lustre: 100352:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3670.493970] Lustre: 100352:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 3670.498586] Lustre: 100352:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3670.509313] Lustre: 100352:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3670.514822] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3670.531220] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3670.561899] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3670.675791] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 3670.679760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1601) [ 3675.623493] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3676.012270] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3700.226075] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3702.430869] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 15:10:06 (1780341006) [ 3713.201912] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3716.514979] Lustre: Failing over lustre-MDT0000 [ 3716.578023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3716.586380] Lustre: Skipped 3 previous similar messages [ 3716.811529] Lustre: server umount lustre-MDT0000 complete [ 3732.640861] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3732.643914] LDISKFS-fs (dm-0): recovery complete [ 3732.655811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.213428] Lustre: lustre-MDT0000: Aborting client recovery [ 3733.216371] LustreError: 102268:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3733.231438] Lustre: 102302:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3733.238431] Lustre: 102302:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 3733.248359] Lustre: 102302:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3733.263123] Lustre: 102302:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3733.283690] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3733.289983] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3733.303914] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3733.349235] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 3733.354590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1633) [ 3738.605799] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3738.635287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3738.648577] Lustre: Skipped 42 previous similar messages [ 3738.802162] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3765.364871] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 15:11:09 (1780341069) [ 3817.261928] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3819.436725] Lustre: Failing over lustre-MDT0000 [ 3819.642277] Lustre: server umount lustre-MDT0000 complete [ 3848.297900] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3848.304430] LDISKFS-fs (dm-0): recovery complete [ 3848.319815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.471514] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cad42d43800 x1866819081186688/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3861.520642] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 72 previous similar messages [ 3866.983918] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3867.295158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3867.299196] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3876.393311] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3879.431565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3906.085486] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 15:13:30 (1780341210) [ 3940.525517] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3953.026571] Lustre: Failing over lustre-MDT0000 [ 3953.072087] LustreError: 3670:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cae6f595880 x1866819081350784/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3953.340447] Lustre: server umount lustre-MDT0000 complete [ 3954.147173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.168083] Lustre: Skipped 36 previous similar messages [ 3970.531690] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780341261/real 1780341261] req@ffff8cae7c66f480 x1866819081351680/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780341277 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3970.588825] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 3970.600558] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3970.613486] LustreError: Skipped 7 previous similar messages [ 3980.018762] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3980.023519] LDISKFS-fs (dm-0): recovery complete [ 3980.036750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.616910] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3981.631235] Lustre: Skipped 2 previous similar messages [ 3986.378540] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3991.060603] Lustre: lustre-MDT0000: Recovery over after 0:10, of 2 clients 2 recovered and 0 were evicted. [ 3991.077913] Lustre: Skipped 2 previous similar messages [ 3991.141597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3991.144918] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3995.465637] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3997.467256] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4021.431784] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 15:15:26 (1780341326) [ 4023.634270] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4024.737740] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4024.744223] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4032.099560] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 15:15:37 (1780341337) [ 4063.038317] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4076.729310] Lustre: Failing over lustre-OST0000 [ 4076.835780] Lustre: server umount lustre-OST0000 complete [ 4077.024554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4077.031081] LustreError: Skipped 7 previous similar messages [ 4093.920819] LustreError: 42146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.942376] LustreError: 42146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 163 previous similar messages [ 4100.612435] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 4100.617907] LDISKFS-fs (dm-2): recovery complete [ 4100.629994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.532226] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4171.369714] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 15:17:55 (1780341475) [ 4181.281572] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4184.724830] Lustre: Failing over lustre-MDT0000 [ 4185.143166] Lustre: server umount lustre-MDT0000 complete [ 4208.778108] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4208.780143] LDISKFS-fs (dm-0): recovery complete [ 4208.796032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4215.289193] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4215.295529] Lustre: Skipped 17 previous similar messages [ 4220.477717] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4220.516684] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 4220.517327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 4229.115382] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4231.257984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4236.768714] LustreError: 111458:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4236.770980] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4237.792963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 4252.143117] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 15:19:16 (1780341556) [ 4258.109373] LustreError: 14697:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4263.396704] LustreError: 14697:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4263.405631] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnecting [ 4263.454922] LustreError: 42144:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 4265.499931] LustreError: 6528:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4270.559409] LustreError: 6528:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4270.570675] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnecting [ 4272.573304] LustreError: 14697:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4277.731413] LustreError: 14697:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4277.739394] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnecting [ 4279.846429] LustreError: 6529:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4284.895380] LustreError: 6529:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4286.976941] LustreError: 14697:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4292.063296] LustreError: 14697:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4292.078552] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnecting [ 4292.091420] Lustre: Skipped 1 previous similar message [ 4301.362112] LustreError: 8559:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4301.379138] LustreError: 8559:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4306.401151] LustreError: 8559:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4306.417668] LustreError: 8559:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4313.567541] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnecting [ 4313.580171] Lustre: Skipped 2 previous similar messages [ 4322.743167] LustreError: 6528:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4322.751605] LustreError: 6528:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4327.912584] LustreError: 6528:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4327.927394] LustreError: 6528:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4336.817167] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 15:20:41 (1780341641) [ 4338.598091] LustreError: 82961:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4348.212204] Lustre: lustre-MDT0000: Export ffff8cae45996000 already connecting from 192.168.204.38@tcp [ 4351.210339] Lustre: lustre-MDT0000: Export ffff8cae45996000 already connecting from 192.168.204.38@tcp [ 4353.279913] Lustre: lustre-MDT0000: Export ffff8cae45996000 already connecting from 192.168.204.38@tcp [ 4358.442181] Lustre: lustre-MDT0000: Export ffff8cae45996000 already connecting from 192.168.204.38@tcp [ 4363.567194] Lustre: lustre-MDT0000: Export ffff8cae45996000 already connecting from 192.168.204.38@tcp [ 4363.574100] Lustre: Skipped 1 previous similar message [ 4373.802075] Lustre: lustre-MDT0000: Export ffff8cae45996000 already connecting from 192.168.204.38@tcp [ 4373.812522] Lustre: Skipped 1 previous similar message [ 4378.615806] LustreError: 82961:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4378.628152] Lustre: 82961:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8cae6f558a80 x1866819059929728/t0(0) o38->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:0/0 lens 520/416 e 0 to 0 dl 1780341665 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4378.925460] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnecting [ 4378.938776] Lustre: Skipped 3 previous similar messages [ 4378.943343] LustreError: 6529:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4404.527103] Lustre: lustre-MDT0000: Export ffff8cae45996000 already connecting from 192.168.204.38@tcp [ 4418.999209] LustreError: 6529:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4419.006177] Lustre: 6529:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8cae7d823b80 x1866819059934336/t0(0) o38->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:0/0 lens 520/416 e 0 to 0 dl 1780341705 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4419.884988] LustreError: 8559:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4445.483093] Lustre: lustre-MDT0000: Export ffff8cae45996000 already connecting from 192.168.204.38@tcp [ 4445.492211] Lustre: Skipped 3 previous similar messages [ 4459.887435] LustreError: 8559:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4459.893316] Lustre: 8559:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8cae6ef99500 x1866819059937536/t0(0) o38->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:0/0 lens 520/416 e 0 to 0 dl 1780341746 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4460.844657] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnecting [ 4460.858583] Lustre: Skipped 1 previous similar message [ 4460.868650] LustreError: 6528:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4500.951707] LustreError: 6528:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4500.961931] Lustre: 6528:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8cae5ce2f800 x1866819059940736/t0(0) o38->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:0/0 lens 520/416 e 0 to 0 dl 1780341787 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4501.290638] LustreError: 14697:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4525.867222] Lustre: lustre-MDT0000: Export ffff8cae45996000 already connecting from 192.168.204.38@tcp [ 4525.878531] Lustre: Skipped 6 previous similar messages [ 4541.311129] LustreError: 14697:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4541.318138] Lustre: 14697:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8cae5fb66300 x1866819059943808/t0(0) o38->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:0/0 lens 520/416 e 0 to 0 dl 1780341828 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4546.345704] LustreError: 8559:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4561.536138] LustreError: 8559:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout interrupted [ 4564.994355] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 15:24:30 (1780341870) [ 4570.331833] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4573.167494] Lustre: Failing over lustre-MDT0000 [ 4573.348074] Lustre: server umount lustre-MDT0000 complete [ 4573.666252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4573.682088] Lustre: Skipped 8 previous similar messages [ 4583.028175] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4583.030839] LDISKFS-fs (dm-0): recovery complete [ 4583.040825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4583.150780] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.156281] LustreError: Skipped 1 previous similar message [ 4583.277267] Lustre: *** cfs_fail_loc=712, val=0*** [ 4583.282491] LustreError: 42148:0:(service.c:1397:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8cae459d9c00 x1866819081865472/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4583.339316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4583.342128] Lustre: Skipped 5 previous similar messages [ 4583.372984] Lustre: lustre-MDT0000: Aborting client recovery [ 4583.375551] LustreError: 116195:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4583.380672] Lustre: 116228:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4583.391728] Lustre: 116228:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 4583.402845] Lustre: 116228:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4583.413207] Lustre: 116228:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4583.422437] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4583.431211] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4583.442496] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4583.478048] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 4583.478227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 4586.273868] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4588.523156] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4588.535354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4588.543343] Lustre: Skipped 19 previous similar messages [ 4599.412467] Lustre: Failing over lustre-MDT0000 [ 4599.587719] Lustre: server umount lustre-MDT0000 complete [ 4615.838366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4618.031150] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4618.038905] Lustre: Skipped 2 previous similar messages [ 4618.998336] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4621.308687] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4621.315116] Lustre: Skipped 2 previous similar messages [ 4621.348954] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 4621.349627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 4624.489285] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4625.562496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4631.380986] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 15:25:36 (1780341936) [ 4631.551183] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnecting [ 4631.555510] Lustre: Skipped 2 previous similar messages [ 4637.195888] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 15:25:42 (1780341942) [ 4637.909989] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4637.918361] LustreError: 6537:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae72c1f480 x1866819060004736/t0(0) o700->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:615/0 lens 264/248 e 0 to 0 dl 1780341955 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4656.456358] Lustre: Failing over lustre-MDT0000 [ 4656.653859] Lustre: server umount lustre-MDT0000 complete [ 4672.411872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4675.728090] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4678.166825] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 4678.167153] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 4681.343622] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4682.543750] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4689.441917] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 15:26:34 (1780341994) [ 4691.215943] Lustre: Failing over lustre-OST0000 [ 4691.325973] Lustre: server umount lustre-OST0000 complete [ 4691.945111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4691.952030] LustreError: Skipped 5 previous similar messages [ 4695.342724] LustreError: 42142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.354224] LustreError: 42142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 88 previous similar messages [ 4707.018793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4711.450907] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4716.921686] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4718.024599] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4787.293652] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 15:28:12 (1780342092) [ 4792.625609] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4794.412341] Lustre: Failing over lustre-MDT0000 [ 4794.612741] Lustre: server umount lustre-MDT0000 complete [ 4813.182787] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4813.185883] LDISKFS-fs (dm-0): recovery complete [ 4813.196034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.389636] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4819.257620] Lustre: *** cfs_fail_loc=216, val=0*** [ 4819.257634] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4819.260619] LustreError: 121991:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 4820.320710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4886.230873] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 15:29:51 (1780342191) [ 4888.042767] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4888.047095] Lustre: Skipped 3 previous similar messages [ 4898.038542] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 15:30:03 (1780342203) [ 4900.148612] Lustre: Failing over lustre-MDT0000 [ 4900.353581] Lustre: server umount lustre-MDT0000 complete [ 4916.332900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4916.614505] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4916.622794] Lustre: Skipped 7 previous similar messages [ 4919.474480] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4921.841130] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4921.844447] LustreError: 123557:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae459d9c00 x1866819060126592/t0(0) o101->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:184/0 lens 328/344 e 0 to 0 dl 1780342279 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4978.474980] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnected, waiting for 2 clients in recovery for 1:06 [ 4978.501341] Lustre: 123557:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4978.545170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 4978.545402] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 4981.267317] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4982.290698] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4988.965444] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 15:31:34 (1780342294) [ 4990.734924] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4995.460618] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4996.904921] Lustre: Failing over lustre-MDT0000 [ 4997.130410] Lustre: server umount lustre-MDT0000 complete [ 5015.007458] Lustre: 3670:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780342305/real 1780342305] req@ffff8cae72c1ce00 x1866819082107904/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780342321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5015.025495] Lustre: 3670:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5015.059809] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5015.062414] LDISKFS-fs (dm-0): recovery complete [ 5015.068316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5027.008525] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5029.887226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 5029.890214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 5032.165333] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5033.247869] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5038.802577] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 15:32:24 (1780342344) [ 5039.532163] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5045.086464] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5046.179255] Lustre: Failing over lustre-MDT0000 [ 5046.322706] Lustre: server umount lustre-MDT0000 complete [ 5064.433920] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5064.435626] LDISKFS-fs (dm-0): recovery complete [ 5064.441925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5067.249565] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5069.833993] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 5069.838835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3169) [ 5072.369154] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5073.454398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5079.659701] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 15:33:05 (1780342385) [ 5080.369131] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5087.350396] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5088.663655] Lustre: Failing over lustre-MDT0000 [ 5088.843524] Lustre: server umount lustre-MDT0000 complete [ 5107.603486] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5107.606221] LDISKFS-fs (dm-0): recovery complete [ 5107.619684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.895640] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae45a47480 x1866819082173696/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5116.914948] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 5119.727408] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5122.581228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 5122.582913] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3169) [ 5129.078338] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 15:33:54 (1780342434) [ 5130.851806] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5130.853882] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5130.862615] LustreError: 6530:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae5fb64700 x1866819060180608/t257698037777(0) o35->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:353/0 lens 392/456 e 0 to 0 dl 1780342448 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5132.710887] Lustre: Failing over lustre-MDT0000 [ 5132.774252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5132.780663] Lustre: Skipped 3 previous similar messages [ 5132.934454] Lustre: server umount lustre-MDT0000 complete [ 5148.433921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5151.313488] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5153.795141] Lustre: 41868:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cae5f93e300 x1866819060180608/t257698037777(0) o35->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:376/0 lens 392/456 e 0 to 0 dl 1780342471 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5153.807382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3233) [ 5153.807426] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 5156.182775] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5157.295532] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5163.266435] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 15:34:28 (1780342468) [ 5163.974445] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5163.984409] LustreError: 6528:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae5fb66300 x1866819060193152/t261993005072(0) o36->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:430/0 lens 504/448 e 0 to 0 dl 1780342525 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5170.521726] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5171.658552] Lustre: Failing over lustre-MDT0000 [ 5171.879495] Lustre: server umount lustre-MDT0000 complete [ 5174.240279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5174.253808] Lustre: Skipped 42 previous similar messages [ 5190.623181] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5190.639046] LustreError: Skipped 8 previous similar messages [ 5190.657604] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5190.660177] LDISKFS-fs (dm-0): recovery complete [ 5190.666220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.867120] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cad42d43800 x1866819082227328/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5201.033641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5201.043477] Lustre: Skipped 9 previous similar messages [ 5203.751992] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5206.501957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5206.506552] Lustre: Skipped 38 previous similar messages [ 5206.553519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 5206.557742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3265) [ 5206.572132] Lustre: 82961:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cae5ce2f480 x1866819060193152/t261993005072(0) o36->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:473/0 lens 504/2880 e 0 to 0 dl 1780342568 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5209.218829] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5210.435310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5217.049898] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 15:35:22 (1780342522) [ 5217.833161] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5217.837503] LustreError: 6527:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae72a2b480 x1866819060209280/t266287972368(0) o36->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:484/0 lens 504/448 e 0 to 0 dl 1780342579 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5219.491700] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5224.988730] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5226.512760] Lustre: Failing over lustre-MDT0000 [ 5226.674392] Lustre: server umount lustre-MDT0000 complete [ 5244.526643] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5244.528832] LDISKFS-fs (dm-0): recovery complete [ 5244.537395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.210181] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5247.218162] Lustre: Skipped 9 previous similar messages [ 5247.320401] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5250.034861] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5250.043107] Lustre: Skipped 9 previous similar messages [ 5250.071155] Lustre: 6527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cae5f93d180 x1866819060209280/t266287972368(0) o36->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:516/0 lens 504/2880 e 0 to 0 dl 1780342611 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5250.076993] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 5250.077105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3297) [ 5250.100436] Lustre: 6527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5257.053782] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 15:36:02 (1780342562) [ 5257.784495] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5257.788957] Lustre: Skipped 1 previous similar message [ 5257.791676] LustreError: 82961:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae474d9c00 x1866819060222720/t270582939664(0) o36->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:524/0 lens 504/448 e 0 to 0 dl 1780342619 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5257.808800] LustreError: 82961:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 1 previous similar message [ 5259.326675] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5259.330523] Lustre: Skipped 1 previous similar message [ 5264.947386] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5266.140569] Lustre: Failing over lustre-MDT0000 [ 5266.277741] Lustre: server umount lustre-MDT0000 complete [ 5284.987593] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5284.989600] LDISKFS-fs (dm-0): recovery complete [ 5284.995980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5288.215586] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5290.510749] Lustre: 6527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cae45a44000 x1866819060222720/t270582939664(0) o36->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:557/0 lens 504/2880 e 0 to 0 dl 1780342652 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5290.513512] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3297) [ 5290.513669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 5297.360294] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 15:36:42 (1780342602) [ 5298.183562] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5299.749457] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5299.752470] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5299.770472] LustreError: 6530:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae5f984000 x1866819060235904/t274877906960(0) o35->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:522/0 lens 392/456 e 0 to 0 dl 1780342617 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5305.688855] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5307.018984] Lustre: Failing over lustre-MDT0000 [ 5307.197059] Lustre: server umount lustre-MDT0000 complete [ 5310.260307] LustreError: 82961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5310.271588] LustreError: 82961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 211 previous similar messages [ 5310.944492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5310.949282] LustreError: Skipped 7 previous similar messages [ 5325.195413] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5325.197913] LDISKFS-fs (dm-0): recovery complete [ 5325.203322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.181700] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5330.946780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 5330.946782] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3329) [ 5330.961738] Lustre: 41868:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cad44a81c00 x1866819060235904/t274877906960(0) o35->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:553/0 lens 392/456 e 0 to 0 dl 1780342648 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5338.614407] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 15:37:23 (1780342643) [ 5339.305059] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 5339.312252] LustreError: 8559:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cad42cb4700 x1866819060246912/t279172874255(0) o101->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:606/0 lens 664/608 e 0 to 0 dl 1780342701 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5399.346207] Lustre: 6529:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cae509c5180 x1866819060246912/t279172874255(0) o101->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:666/0 lens 664/3488 e 0 to 0 dl 1780342761 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5403.762958] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 15:38:29 (1780342709) [ 5408.831684] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5410.050510] Lustre: Failing over lustre-MDT0000 [ 5410.239915] Lustre: server umount lustre-MDT0000 complete [ 5428.898707] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5428.901295] LDISKFS-fs (dm-0): recovery complete [ 5428.912412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5438.431857] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cad45421880 x1866819082362368/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5441.474895] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5444.112029] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 5444.112380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3393) [ 5446.554710] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5447.601976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5464.227811] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 15:39:29 (1780342769) [ 5470.325085] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5471.656939] Lustre: Failing over lustre-MDT0000 [ 5471.801777] Lustre: server umount lustre-MDT0000 complete [ 5490.649944] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5490.652312] LDISKFS-fs (dm-0): recovery complete [ 5490.658927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.823461] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5506.116651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3425) [ 5506.122783] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 5509.547293] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5510.749709] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5514.586737] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5524.002074] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 15:40:29 (1780342829) [ 5606.499687] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5609.349590] Lustre: Failing over lustre-MDT0000 [ 5610.003197] Lustre: server umount lustre-MDT0000 complete [ 5629.919395] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780342920/real 1780342920] req@ffff8cad44544380 x1866819082466432/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780342936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5629.959143] Lustre: 3667:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5633.581107] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5633.584897] LDISKFS-fs (dm-0): recovery complete [ 5633.599315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5639.469985] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5639.476927] Lustre: Skipped 10 previous similar messages [ 5643.648497] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5645.273012] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 5645.273807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 5651.644835] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5653.992226] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5715.534073] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 15:43:40 (1780343020) [ 5724.397427] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5726.274086] Lustre: Failing over lustre-MDT0000 [ 5726.490150] Lustre: server umount lustre-MDT0000 complete [ 5747.696984] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5747.699955] LDISKFS-fs (dm-0): recovery complete [ 5747.705798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5757.409170] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae5cdf8000 x1866819083174272/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5757.448434] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 5762.213378] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5763.185367] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 5763.185367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 5770.267846] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5772.167760] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5780.547619] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 5782.136196] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5788.350898] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 15:44:53 (1780343093) [ 5796.220425] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5856.070976] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnecting [ 5856.084563] Lustre: Skipped 1 previous similar message [ 5859.084665] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5859.086669] Lustre: Skipped 1 previous similar message [ 5859.094729] LustreError: 6527:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae6f594e00 x1866819063056256/t296352743435(0) o36->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:370/0 lens 66040/440 e 0 to 0 dl 1780343220 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5919.571934] Lustre: 6529:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cae49819050 x1866819063056256/t296352743435(0) o36->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:431/0 lens 66040/440 e 0 to 0 dl 1780343281 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5931.552793] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5933.738479] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 15:47:18 (1780343238) [ 5948.396211] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5951.552934] Lustre: Failing over lustre-MDT0000 [ 5951.885624] Lustre: server umount lustre-MDT0000 complete [ 5952.481518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5952.485349] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5952.486078] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.486085] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 146 previous similar messages [ 5952.492636] LustreError: Skipped 3 previous similar messages [ 5952.540855] Lustre: Skipped 30 previous similar messages [ 5968.863236] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5968.875648] LustreError: Skipped 7 previous similar messages [ 5975.056425] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5975.059651] LDISKFS-fs (dm-0): recovery complete [ 5975.067833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5978.429502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.438628] Lustre: Skipped 7 previous similar messages [ 5980.971684] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5980.986806] Lustre: Skipped 6 previous similar messages [ 5983.662566] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5983.724540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5983.737583] Lustre: Skipped 31 previous similar messages [ 5984.834669] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5984.842514] Lustre: Skipped 6 previous similar messages [ 5984.880679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 5984.882909] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 5991.164144] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5992.987511] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6003.644255] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 15:48:28 (1780343308) [ 6027.414991] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6040.389106] Lustre: Failing over lustre-OST0000 [ 6040.542430] Lustre: server umount lustre-OST0000 complete [ 6063.906582] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6063.908768] LDISKFS-fs (dm-2): recovery complete [ 6063.934025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6070.511068] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6084.952739] Lustre: Failing over lustre-OST0000 [ 6085.095147] Lustre: server umount lustre-OST0000 complete [ 6103.677779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6109.943712] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6117.772366] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6119.587675] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6158.961045] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 15:51:04 (1780343464) [ 6161.617116] Lustre: Failing over lustre-MDT0000 [ 6161.910467] Lustre: server umount lustre-MDT0000 complete [ 6179.997903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.631681] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6195.786197] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 6195.786798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 6208.881062] Lustre: Failing over lustre-MDT0000 [ 6209.089171] Lustre: server umount lustre-MDT0000 complete [ 6226.401040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.641781] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f367558a [ 6236.660316] Lustre: Skipped 1 previous similar message [ 6240.388675] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6242.348938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 6242.351337] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 6246.840775] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6248.419921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6256.354491] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 15:52:41 (1780343561) [ 6270.083912] Lustre: Failing over lustre-OST0000 [ 6270.150355] Lustre: server umount lustre-OST0000 complete [ 6287.369707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6287.531431] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6287.537660] Lustre: Skipped 6 previous similar messages [ 6292.178691] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6299.902444] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6301.431912] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6310.237342] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 15:53:35 (1780343615) [ 6312.130125] Lustre: Failing over lustre-MDT0000 [ 6312.328534] Lustre: server umount lustre-MDT0000 complete [ 6320.123142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6320.192223] Lustre: *** cfs_fail_loc=605, val=0*** [ 6320.194453] LustreError: 157404:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc095b490 failed: rc = -95 [ 6320.200714] LustreError: 157404:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 6320.204981] LustreError: 157404:0:(obd_mount.c:250:lustre_start_simple()) MGS setup error -95 [ 6320.209853] LustreError: 157404:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 6320.213339] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 6320.217839] LustreError: 157404:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 6320.227206] Lustre: server umount lustre-MDT0000 complete [ 6320.229259] LustreError: 157404:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 6326.708705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.666443] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6332.451766] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 6332.452331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 6337.349550] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 15:54:02 (1780343642) [ 6343.060089] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6345.613947] Lustre: Failing over lustre-MDT0000 [ 6345.862401] Lustre: server umount lustre-MDT0000 complete [ 6364.127981] Lustre: 3669:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780343654/real 1780343654] req@ffff8cae7cac6d80 x1866819083636224/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780343670 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6364.159705] Lustre: 3669:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6365.971814] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6365.974973] LDISKFS-fs (dm-0): recovery complete [ 6365.983766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6374.372327] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f3676367 [ 6375.238415] Lustre: *** cfs_fail_loc=707, val=0*** [ 6377.781982] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6435.644757] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 6435.689488] Lustre: 159565:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 6435.700391] Lustre: 159565:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 1 previous similar message [ 6436.121555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 6436.121730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 6440.051386] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6441.539458] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6450.373064] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 15:55:55 (1780343755) [ 6478.929223] LustreError: 8559:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8cae459d8e00 x1866819063954304/t0(0) o101->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:191/0 lens 664/0 e 0 to 0 dl 1780343796 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6478.946233] LustreError: 8559:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 6489.999294] LustreError: 8559:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6490.015170] LustreError: 41868:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8cae5ce2ca80 x1866819063955328/t0(0) o35->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:202/0 lens 392/0 e 0 to 0 dl 1780343807 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6491.746885] LustreError: 6528:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8cad45551500 x1866819063961216/t0(0) o101->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:243/0 lens 576/0 e 0 to 0 dl 1780343848 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 6491.757199] LustreError: 6528:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 6495.716645] LustreError: 42148:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8cae72a2bb80 x1866819083705600/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:208/0 lens 544/0 e 0 to 0 dl 1780343813 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 6495.744027] LustreError: 42148:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 20 previous similar messages [ 6506.512245] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 15:56:51 (1780343811) [ 6534.481110] LustreError: 8412:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 6545.511132] LustreError: 8412:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 6552.503411] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 15:57:37 (1780343857) [ 6578.459970] LustreError: 6529:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8cae7c66fb80 x1866819063979264/t0(0) o101->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:291/0 lens 576/0 e 0 to 0 dl 1780343896 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6578.473501] LustreError: 6529:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 6583.575653] LustreError: 6529:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6588.898325] LustreError: 8565:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8cae6f597480 x1866819083747584/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:301/0 lens 544/0 e 0 to 0 dl 1780343906 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 6588.925017] LustreError: 8565:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 101 previous similar messages [ 6595.455288] LustreError: 17314:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6611.682335] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 15:58:37 (1780343917) [ 6712.043886] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 16:00:17 (1780344017) [ 6738.389612] LustreError: 17314:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8cae5cfef100 x1866819064059392/t0(0) o101->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:451/0 lens 576/0 e 0 to 0 dl 1780344056 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6738.414302] LustreError: 17314:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 117 previous similar messages [ 6738.425647] LustreError: 17314:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6738.433033] LustreError: 17314:0:(service.c:2562:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 6738.855188] LustreError: 17314:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6754.665266] LustreError: 6530:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6754.685544] LustreError: 6530:0:(service.c:2562:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 6755.111159] LustreError: 6530:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6755.119247] LustreError: 6530:0:(service.c:2562:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 6770.504092] LustreError: 6528:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff8cae6ef99500 x1866819064078592/t0(0) o36->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:483/0 lens 504/0 e 0 to 0 dl 1780344088 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 6770.519287] LustreError: 6528:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 78 previous similar messages [ 6786.746228] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 16:01:32 (1780344092) [ 6817.566780] Lustre: DEBUG MARKER: phase 2 [ 6825.501060] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 16:02:10 (1780344130) [ 6903.276414] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 16:03:28 (1780344208) [ 6904.389825] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6905.662702] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 16:03:30 (1780344210) [ 6909.883561] Lustre: DEBUG MARKER: Started rundbench load pid=134010 ... [ 6916.622894] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6918.872300] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6920.183733] Lustre: Failing over lustre-MDT0000 [ 6920.410078] Lustre: server umount lustre-MDT0000 complete [ 6921.030518] LustreError: 8559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6921.043266] LustreError: 8559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 170 previous similar messages [ 6922.720588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6922.725269] LustreError: Skipped 7 previous similar messages [ 6922.729099] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6922.736149] Lustre: Skipped 25 previous similar messages [ 6938.902364] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6938.912779] LustreError: Skipped 4 previous similar messages [ 6939.237215] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 6939.240060] LDISKFS-fs (dm-0): recovery complete [ 6939.253830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6948.499155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6948.502986] Lustre: Skipped 7 previous similar messages [ 6948.539176] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6948.544600] Lustre: Skipped 2 previous similar messages [ 6950.322411] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6950.330955] Lustre: Skipped 7 previous similar messages [ 6951.230256] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6953.962734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6953.969248] Lustre: Skipped 27 previous similar messages [ 6954.014414] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6954.020649] Lustre: Skipped 7 previous similar messages [ 6954.054775] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 6954.065141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 6956.997767] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6958.269658] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6966.606594] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6968.831663] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6970.160781] Lustre: Failing over lustre-MDT0001 [ 6970.183442] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.38@tcp (stopping) [ 6970.394960] Lustre: server umount lustre-MDT0001 complete [ 6988.396666] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 6988.398218] LDISKFS-fs (dm-1): recovery complete [ 6988.406801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6991.265354] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6994.389277] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:563 to 0x2c0000400:609) [ 6994.390415] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:581 to 0x280000400:609) [ 6996.970884] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6998.069266] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7005.347111] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7007.482377] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 7008.750817] Lustre: Failing over lustre-MDT0000 [ 7008.971045] Lustre: server umount lustre-MDT0000 complete [ 7025.632511] Lustre: 3670:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780344316/real 1780344316] req@ffff8cad4c854a80 x1866819084333312/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780344332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7025.653211] Lustre: 3670:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7026.610530] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7026.614570] LDISKFS-fs (dm-0): recovery complete [ 7026.621973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.663892] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7041.564732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 7041.565462] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 7044.421318] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7045.778618] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7057.519784] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 16:06:03 (1780344363) [ 7182.786465] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7193.881087] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 7194.843168] Lustre: Failing over lustre-MDT0000 [ 7194.869296] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.38@tcp (stopping) [ 7195.078802] Lustre: server umount lustre-MDT0000 complete [ 7212.100716] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7212.103933] LDISKFS-fs (dm-0): recovery complete [ 7212.111660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7224.148817] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7229.762748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5460 to 0x2c0000401:5505) [ 7229.763303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5492 to 0x280000401:5537) [ 7232.198419] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7233.164782] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7359.023861] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7370.309866] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 7371.366658] Lustre: Failing over lustre-MDT0000 [ 7371.380204] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.38@tcp (stopping) [ 7371.632206] Lustre: server umount lustre-MDT0000 complete [ 7388.758907] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7388.761990] LDISKFS-fs (dm-0): recovery complete [ 7388.769632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7391.212536] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7396.876156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5948 to 0x280000401:5985) [ 7396.876161] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5917 to 0x2c0000401:5953) [ 7398.997467] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7400.046380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7428.046907] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 16:12:13 (1780344733) [ 7552.852971] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7564.227027] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 7565.483991] Lustre: Failing over lustre-MDT0001 [ 7565.489918] LustreError: lustre-MDT0001-osp-MDT0000: operation ldlm_enqueue to node 0@lo failed: rc = -19 [ 7565.493926] LustreError: Skipped 4 previous similar messages [ 7565.497923] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7565.504740] Lustre: Skipped 16 previous similar messages [ 7565.508427] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7565.718338] Lustre: server umount lustre-MDT0001 complete [ 7568.352286] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7568.365405] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 128 previous similar messages [ 7582.703579] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7582.705735] LDISKFS-fs (dm-1): recovery complete [ 7582.717288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7582.899506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7582.903587] Lustre: Skipped 4 previous similar messages [ 7582.925790] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7582.930604] Lustre: Skipped 4 previous similar messages [ 7584.113339] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7584.118497] Lustre: Skipped 4 previous similar messages [ 7584.754374] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7588.323100] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7588.325792] Lustre: Skipped 18 previous similar messages [ 7589.223694] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7589.227295] Lustre: Skipped 4 previous similar messages [ 7589.247882] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1582 to 0x280000400:1601) [ 7589.247926] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1581 to 0x2c0000400:1601) [ 7591.246650] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7592.212815] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7717.880627] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7728.858629] Lustre: DEBUG MARKER: test_70d fail mds2 2 times [ 7729.801756] Lustre: Failing over lustre-MDT0001 [ 7729.814331] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.38@tcp (stopping) [ 7729.817038] Lustre: Skipped 1 previous similar message [ 7730.002893] Lustre: server umount lustre-MDT0001 complete [ 7745.508310] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7745.510200] LDISKFS-fs (dm-1): recovery complete [ 7745.518607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.481111] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7816.169807] Lustre: 177853:0:(ldlm_lib.c:2932:target_recovery_thread()) too long recovery - read logs [ 7816.177374] LustreError: dumping log to /tmp/lustre-log.1780345123.177853 [ 7816.266637] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1775 to 0x280000400:1793) [ 7816.266769] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1775 to 0x2c0000400:1793) [ 7818.027877] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7818.835438] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7823.094323] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 16:18:48 (1780345128) [ 7947.303408] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7958.286693] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 7959.123617] Lustre: Failing over lustre-MDT0000 [ 7959.169566] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.38@tcp (stopping) [ 7959.376364] Lustre: server umount lustre-MDT0000 complete [ 7975.057691] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7975.062883] LDISKFS-fs (dm-0): recovery complete [ 7975.073878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.148919] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7975.156870] LustreError: Skipped 3 previous similar messages [ 7976.939863] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7984.326051] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7302 to 0x2c0000401:7329) [ 7984.326284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7334 to 0x280000401:7361) [ 7985.963990] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7986.789536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8112.181228] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8123.173986] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 8123.991511] Lustre: Failing over lustre-MDT0000 [ 8124.038163] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.38@tcp (stopping) [ 8124.262128] Lustre: server umount lustre-MDT0000 complete [ 8139.848574] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8139.850702] LDISKFS-fs (dm-0): recovery complete [ 8139.855769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8141.808051] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8149.212221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8382 to 0x280000401:8417) [ 8149.221960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8350 to 0x2c0000401:8385) [ 8150.910278] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8151.656422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8156.069326] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 16:24:21 (1780345461) [ 8163.008826] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8164.665289] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 8165.419822] Lustre: Failing over lustre-OST0000 [ 8165.467414] Lustre: server umount lustre-OST0000 complete [ 8165.856812] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8165.864954] Lustre: Skipped 14 previous similar messages [ 8170.976052] LustreError: 172962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8170.982079] LustreError: 172962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 58 previous similar messages [ 8180.304468] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8180.305965] LDISKFS-fs (dm-2): recovery complete [ 8180.311581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8182.687447] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8186.372497] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8187.095521] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8196.583387] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8198.257611] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 8198.971532] Lustre: Failing over lustre-OST0000 [ 8199.020640] Lustre: server umount lustre-OST0000 complete [ 8213.688878] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8213.691046] LDISKFS-fs (dm-2): recovery complete [ 8213.696855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8213.780981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8213.785300] Lustre: Skipped 4 previous similar messages [ 8213.790106] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 8213.792535] Lustre: Skipped 4 previous similar messages [ 8215.084719] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8215.089067] Lustre: Skipped 4 previous similar messages [ 8215.717076] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8215.717107] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8215.720128] Lustre: Skipped 4 previous similar messages [ 8215.726496] Lustre: Skipped 15 previous similar messages [ 8215.935194] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8219.506485] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8220.218091] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8226.091626] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 16:25:31 (1780345531) [ 8349.543611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8352.790982] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8363.743921] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 8364.576495] Lustre: Failing over lustre-MDT0000 [ 8364.593367] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.38@tcp (stopping) [ 8364.775794] Lustre: server umount lustre-MDT0000 complete [ 8366.297896] LustreError: 167674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345673 with bad export cookie 9786937278492443240 [ 8366.300479] Lustre: Failing over lustre-MDT0001 [ 8366.302482] LustreError: 167674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8366.518270] Lustre: server umount lustre-MDT0001 complete [ 8382.004670] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8382.007483] LDISKFS-fs (dm-1): recovery complete [ 8382.016807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.040940] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8382.043531] LDISKFS-fs (dm-0): recovery complete [ 8382.048170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8382.114135] LustreError: 188196:0:(llog.c:1646:llog_backup()) MGC192.168.204.138@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8382.118904] Lustre: 188196:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.204.138@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8383.967289] Lustre: 3669:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780345674/real 1780345674] req@ffff8cad4993c700 x1866819100919552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780345690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8383.985055] Lustre: 3669:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 8391.729279] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 8391.743128] LustreError: Skipped 3 previous similar messages [ 8396.456973] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8396.744336] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8397.663422] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1801 to 0x280000400:1825) [ 8397.670337] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1801 to 0x2c0000400:1825) [ 8406.383650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8460 to 0x280000401:8481) [ 8406.390379] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8428 to 0x2c0000401:8449) [ 8410.459785] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8411.857927] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8414.834459] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8426.715908] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 16:28:50 (1780345730) [ 8437.422481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8441.239148] Lustre: Failing over lustre-MDT0000 [ 8441.566387] Lustre: server umount lustre-MDT0000 complete [ 8459.743941] Lustre: 3668:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780345750/real 1780345750] req@ffff8cad452f9500 x1866819101735936/t0(0) o400->MGC192.168.204.138@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780345766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8459.830061] Lustre: 3668:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 8468.510433] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8468.515107] LDISKFS-fs (dm-0): recovery complete [ 8468.521757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.731717] LustreError: 190501:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 8469.737988] LustreError: 190501:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8cad44fb2680 x1866819101741952/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1780345776 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8469.751529] LustreError: 190501:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 8469.985132] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f39c6702 [ 8470.339213] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 8474.699246] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8485.700721] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 8485.785777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8460 to 0x280000401:8513) [ 8485.786582] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8451 to 0x2c0000401:8481) [ 8488.691472] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8490.070146] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8496.563307] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 16:30:01 (1780345801) [ 8502.173703] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8504.182397] Lustre: Failing over lustre-MDT0000 [ 8504.353964] Lustre: server umount lustre-MDT0000 complete [ 8522.913660] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8522.916204] LDISKFS-fs (dm-0): recovery complete [ 8522.923941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8532.959835] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cae72d8b100 x1866819101782528/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8532.977947] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 6 previous similar messages [ 8533.610927] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 8533.614380] LustreError: 188209:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae72d89c00 x1866819095907200/t343597384112(343597384112) o101->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:736/0 lens 520/664 e 0 to 0 dl 1780345851 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 8536.046348] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8549.200204] Lustre: lustre-MDT0000: Client bee07ca0-5f3b-4b44-b416-9964de954a49 (at 192.168.204.38@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 8549.232268] Lustre: 188208:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cae46df4000 x1866819095907200/t343597384112(343597384112) o101->bee07ca0-5f3b-4b44-b416-9964de954a49@192.168.204.38@tcp:752/0 lens 520/3488 e 0 to 0 dl 1780345867 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 8549.344474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8460 to 0x280000401:8545) [ 8549.345412] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8513) [ 8552.656360] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8554.111941] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8561.176660] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 16:31:06 (1780345866) [ 8564.611100] Lustre: Failing over lustre-OST0000 [ 8564.679328] Lustre: server umount lustre-OST0000 complete [ 8567.117852] Lustre: Failing over lustre-MDT0000 [ 8567.293220] Lustre: server umount lustre-MDT0000 complete [ 8583.084026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.165232] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8583.173743] LustreError: Skipped 4 previous similar messages [ 8586.380454] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8588.020822] Lustre: lustre-MDT0000: Denying connection for new client e3357f50-972d-425a-addd-7995c744c813 (at 192.168.204.38@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 8588.811171] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8545) [ 8597.081418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8601.598913] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8602.615297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8460 to 0x280000401:8577) [ 8610.336129] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 16:31:55 (1780345915) [ 8611.375141] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8611.379106] LustreError: 188213:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae45b6aa00 x1866819101840896/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:59/0 lens 1312/4320 e 0 to 0 dl 1780345929 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8617.948974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8619.245083] Lustre: Failing over lustre-MDT0000 [ 8619.416529] Lustre: server umount lustre-MDT0000 complete [ 8627.679125] Lustre: 188237:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780345918/real 1780345918] req@ffff8cae45b6a300 x1866819101840896/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1780345934 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8627.703179] Lustre: 188237:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8639.280137] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8639.283381] LDISKFS-fs (dm-0): recovery complete [ 8639.291129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8639.456517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 8639.462281] Lustre: Skipped 3 previous similar messages [ 8642.757955] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8644.587557] Lustre: 196324:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 8644.598322] Lustre: 196324:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 53 previous similar messages [ 8644.650440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8609) [ 8644.651266] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8577) [ 8648.701296] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8650.063095] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8657.138109] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 16:32:42 (1780345962) [ 8657.887039] LustreError: 188213:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae72d89180 x1866819101874816/t365072220172(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:105/0 lens 2520/4320 e 0 to 0 dl 1780345975 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8662.614686] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8667.314326] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8668.533797] Lustre: Failing over lustre-MDT0001 [ 8668.537438] LustreError: 197939:0:(ldlm_resource.c:1180:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029441:0x4:0x0].0x0 (ffff8cae454d3800) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8673.948453] Lustre: server umount lustre-MDT0001 complete [ 8690.330279] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8690.332361] LDISKFS-fs (dm-1): recovery complete [ 8690.342365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8692.901181] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8695.843661] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1836 to 0x280000400:1857) [ 8695.843898] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1836 to 0x2c0000400:1857) [ 8695.876560] Lustre: 188209:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cad4abad180 x1866819100668928/t21474837923(0) o36->e3357f50-972d-425a-addd-7995c744c813@192.168.204.38@tcp:168/0 lens 560/2880 e 0 to 0 dl 1780346038 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8699.079719] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8700.399580] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8706.978708] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 16:33:32 (1780346012) [ 8707.797411] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8707.799680] Lustre: Skipped 1 previous similar message [ 8713.231650] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8717.112414] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8717.979148] Lustre: Failing over lustre-MDT0000 [ 8718.196253] Lustre: server umount lustre-MDT0000 complete [ 8733.333671] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8733.335055] LDISKFS-fs (dm-0): recovery complete [ 8733.338473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8735.070158] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8738.836692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8641) [ 8738.836907] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8609) [ 8740.314762] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8741.001501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8743.149882] Lustre: Failing over lustre-MDT0001 [ 8743.267837] Lustre: server umount lustre-MDT0001 complete [ 8758.411611] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8758.414364] LDISKFS-fs (dm-1): recovery complete [ 8758.426535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8760.238670] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8763.895653] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1868 to 0x2c0000400:1889) [ 8763.900025] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1868 to 0x280000400:1889) [ 8765.635450] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8766.538935] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8771.773934] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 16:34:37 (1780346077) [ 8772.379661] LustreError: 188213:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae7cac7800 x1866819101962112/t369367187472(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:220/0 lens 2520/4320 e 0 to 0 dl 1780346090 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8772.393646] LustreError: 188213:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 1 previous similar message [ 8779.610403] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8783.358432] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8784.325850] Lustre: Failing over lustre-MDT0000 [ 8784.355149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.361104] Lustre: Skipped 33 previous similar messages [ 8784.364822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8784.367626] Lustre: Skipped 5 previous similar messages [ 8784.606702] Lustre: server umount lustre-MDT0000 complete [ 8785.722732] LustreError: 200304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8785.732021] LustreError: 200304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 178 previous similar messages [ 8786.363927] LustreError: 167674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346093 with bad export cookie 9786937278493134399 [ 8786.364694] Lustre: Failing over lustre-MDT0001 [ 8786.368549] LustreError: 167674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8792.034337] LustreError: 203866:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cae5cdef100 x1866819101974656/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8792.049297] LustreError: 203866:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8792.225642] Lustre: server umount lustre-MDT0001 complete [ 8807.851907] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8807.854020] LDISKFS-fs (dm-0): recovery complete [ 8807.861696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8807.938176] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8807.940756] LDISKFS-fs (dm-1): recovery complete [ 8807.950637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8812.000503] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f39c9df8 [ 8813.753318] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8813.883554] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8816.353020] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8816.354398] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8816.356499] Lustre: Skipped 11 previous similar messages [ 8816.359789] Lustre: Skipped 36 previous similar messages [ 8817.266735] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8817.272056] Lustre: Skipped 10 previous similar messages [ 8817.291256] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8641) [ 8817.291259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8673) [ 8817.315202] Lustre: 204982:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cad44fe7800 x1866819100740096/t30064771142(0) o36->e3357f50-972d-425a-addd-7995c744c813@192.168.204.38@tcp:290/0 lens 560/2880 e 0 to 0 dl 1780346160 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8817.319729] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1900 to 0x2c0000400:1921) [ 8817.319728] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1900 to 0x280000400:1921) [ 8818.758932] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8819.434126] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8820.062813] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8824.222504] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 16:35:30 (1780346130) [ 8830.563729] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8831.375135] Lustre: Failing over lustre-MDT0000 [ 8831.570877] Lustre: server umount lustre-MDT0000 complete [ 8846.374305] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8846.376656] LDISKFS-fs (dm-0): recovery complete [ 8846.383229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8846.552397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8846.555487] Lustre: Skipped 12 previous similar messages [ 8846.584070] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8846.589664] Lustre: Skipped 12 previous similar messages [ 8848.161144] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8851.963184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8705) [ 8851.963185] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8673) [ 8853.317634] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8853.912693] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8879.913537] Lustre: lustre-MDT0001: Client e3357f50-972d-425a-addd-7995c744c813 (at 192.168.204.38@tcp) reconnecting [ 8879.919229] Lustre: Skipped 1 previous similar message [ 8879.924797] Lustre: 204984:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cae5fb9dc00 x1866819100772608/t34359738438(0) o36->e3357f50-972d-425a-addd-7995c744c813@192.168.204.38@tcp:366/0 lens 560/2880 e 0 to 0 dl 1780346236 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8882.963886] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 16:36:28 (1780346188) [ 8883.387763] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8883.389797] Lustre: Skipped 2 previous similar messages [ 8886.122629] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8886.865356] Lustre: Failing over lustre-MDT0001 [ 8886.961283] Lustre: server umount lustre-MDT0001 complete [ 8901.538502] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8901.540254] LDISKFS-fs (dm-1): recovery complete [ 8901.544606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8903.105963] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8906.742235] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1942 to 0x280000400:1985) [ 8906.742236] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1942 to 0x2c0000400:1985) [ 8907.925233] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8908.477468] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8912.177508] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 16:36:58 (1780346218) [ 8912.527196] LustreError: 207796:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae5fb9c380 x1866819100835456/t38654705732(0) o36->e3357f50-972d-425a-addd-7995c744c813@192.168.204.38@tcp:360/0 lens 560/448 e 0 to 0 dl 1780346230 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8912.534206] LustreError: 207796:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 2 previous similar messages [ 8918.081069] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8920.583475] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8921.232330] Lustre: Failing over lustre-MDT0000 [ 8921.512867] Lustre: server umount lustre-MDT0000 complete [ 8928.041213] Lustre: 205719:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cae4598ce00 x1866819100835456/t38654705732(0) o36->e3357f50-972d-425a-addd-7995c744c813@192.168.204.38@tcp:375/0 lens 560/2880 e 0 to 0 dl 1780346245 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8928.048845] Lustre: 205719:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8935.680812] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8935.683079] LDISKFS-fs (dm-0): recovery complete [ 8935.686919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.110341] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8941.057157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8737) [ 8941.057179] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8705) [ 8942.224618] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8942.772190] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8944.705907] Lustre: Failing over lustre-MDT0001 [ 8944.791606] Lustre: server umount lustre-MDT0001 complete [ 8958.921331] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8958.923072] LDISKFS-fs (dm-1): recovery complete [ 8958.928569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8960.334100] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8964.079550] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1996 to 0x280000400:2017) [ 8964.079570] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1996 to 0x2c0000400:2017) [ 8965.123307] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8965.611707] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8969.022883] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 16:37:54 (1780346274) [ 8974.943512] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8977.571508] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8978.307800] Lustre: Failing over lustre-MDT0000 [ 8978.481267] Lustre: server umount lustre-MDT0000 complete [ 8979.788600] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346286 with bad export cookie 9786937278493148182 [ 8979.789407] Lustre: Failing over lustre-MDT0001 [ 8979.792617] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8979.896213] Lustre: server umount lustre-MDT0001 complete [ 8994.508118] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8994.509931] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8994.510533] LDISKFS-fs (dm-0): recovery complete [ 8994.512207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8994.513094] LDISKFS-fs (dm-1): recovery complete [ 8994.516410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9000.927103] Lustre: 3669:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780346291/real 1780346291] req@ffff8cad4c4b2680 x1866819102139008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780346307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9000.935897] Lustre: 3669:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 9005.023460] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cad4c4b2680 x1866819102140928/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9005.211601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 9005.214498] LustreError: Skipped 14 previous similar messages [ 9006.646780] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9006.729458] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9010.419275] Lustre: 215572:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cad42b03100 x1866819100872960/t42949673028(0) o36->e3357f50-972d-425a-addd-7995c744c813@192.168.204.38@tcp:458/0 lens 560/2880 e 0 to 0 dl 1780346328 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9010.423835] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2028 to 0x280000400:2049) [ 9010.424080] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2028 to 0x2c0000400:2049) [ 9010.436564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8769) [ 9010.436693] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8737) [ 9011.584342] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9012.095093] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9012.575245] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9016.094089] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 16:38:42 (1780346322) [ 9018.955050] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9019.586917] Lustre: Failing over lustre-MDT0001 [ 9025.072119] Lustre: server umount lustre-MDT0001 complete [ 9039.289511] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9039.290842] LDISKFS-fs (dm-1): recovery complete [ 9039.295213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9040.692313] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9044.467183] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2060 to 0x2c0000400:2081) [ 9044.467194] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2060 to 0x280000400:2081) [ 9045.640322] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9046.216621] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9049.718829] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 16:39:15 (1780346355) [ 9052.570480] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9053.172972] Lustre: Failing over lustre-MDT0000 [ 9053.439795] Lustre: server umount lustre-MDT0000 complete [ 9067.332394] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9067.333775] LDISKFS-fs (dm-0): recovery complete [ 9067.337041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9068.791642] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9072.633316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8801) [ 9072.633321] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8769) [ 9073.793373] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9074.349668] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9077.863899] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 16:39:43 (1780346383) [ 9080.630805] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9083.045400] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9083.619217] Lustre: Failing over lustre-MDT0000 [ 9083.813405] Lustre: server umount lustre-MDT0000 complete [ 9097.550254] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9097.551698] LDISKFS-fs (dm-0): recovery complete [ 9097.555754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.865356] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9102.840096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8801) [ 9102.840096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8833) [ 9103.872458] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9104.364971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9106.235925] Lustre: Failing over lustre-MDT0001 [ 9106.317676] Lustre: server umount lustre-MDT0001 complete [ 9120.054305] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9120.055550] LDISKFS-fs (dm-1): recovery complete [ 9120.060136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9121.363909] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9125.359859] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2060 to 0x2c0000400:2113) [ 9125.359905] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2060 to 0x280000400:2113) [ 9126.415986] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9126.883707] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9130.147978] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 16:40:36 (1780346436) [ 9132.851141] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9135.147467] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9135.738737] Lustre: Failing over lustre-MDT0000 [ 9135.974784] Lustre: server umount lustre-MDT0000 complete [ 9137.110758] LustreError: 42147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346444 with bad export cookie 9786937278493158633 [ 9137.112123] Lustre: Failing over lustre-MDT0001 [ 9137.114167] LustreError: 42147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9137.117238] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.38@tcp (stopping) [ 9137.119464] Lustre: Skipped 15 previous similar messages [ 9142.751739] LustreError: 225024:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8cae72d8b480 x1866819102276352/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 9142.759724] LustreError: 225024:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 9142.844953] Lustre: server umount lustre-MDT0001 complete [ 9157.414077] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9157.416433] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9157.416785] LDISKFS-fs (dm-0): recovery complete [ 9157.418676] LDISKFS-fs (dm-1): recovery complete [ 9157.422763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.426222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.493335] LustreError: 226122:0:(llog.c:1646:llog_backup()) MGC192.168.204.138@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 9157.497585] Lustre: 226122:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.204.138@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 9162.207452] LustreError: 3666:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cad492b7800 x1866819102277504/t0(0) o250->MGC192.168.204.138@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9163.619155] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9163.721949] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9168.507409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8865) [ 9168.507417] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8833) [ 9173.617891] Lustre: 226131:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cae5cfb5500 x1866819100948736/t55834574850(0) o36->e3357f50-972d-425a-addd-7995c744c813@192.168.204.38@tcp:646/0 lens 496/2888 e 0 to 0 dl 1780346516 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 9173.625091] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2060 to 0x2c0000400:2145) [ 9173.625096] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2060 to 0x280000400:2145) [ 9173.625787] Lustre: 226131:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 9174.706854] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9175.216738] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9175.694163] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9178.886989] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 16:41:24 (1780346484) [ 9179.222078] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 9179.223712] Lustre: Skipped 6 previous similar messages [ 9179.224820] LustreError: 226876:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff8cae5ceec700 x1866819100962944/t60129542146(0) o36->e3357f50-972d-425a-addd-7995c744c813@192.168.204.38@tcp:666/0 lens 496/456 e 0 to 0 dl 1780346536 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 9179.232067] LustreError: 226876:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 5 previous similar messages [ 9181.801875] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9182.387644] Lustre: Failing over lustre-MDT0000 [ 9182.563024] Lustre: server umount lustre-MDT0000 complete [ 9196.631732] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9196.634148] LDISKFS-fs (dm-0): recovery complete [ 9196.640644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9196.697783] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9196.703720] LustreError: Skipped 9 previous similar messages [ 9198.123163] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9202.198331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8897) [ 9202.198723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8865) [ 9204.328735] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9205.449556] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9234.228542] Lustre: lustre-MDT0001: Client e3357f50-972d-425a-addd-7995c744c813 (at 192.168.204.38@tcp) reconnecting [ 9234.236564] Lustre: Skipped 1 previous similar message [ 9238.566770] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 16:42:24 (1780346544) [ 9244.002588] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9245.186321] Lustre: Failing over lustre-MDT0001 [ 9245.365989] Lustre: server umount lustre-MDT0001 complete [ 9274.887145] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9274.890049] LDISKFS-fs (dm-1): recovery complete [ 9274.941567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9281.236842] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2060 to 0x2c0000400:2177) [ 9281.239218] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2060 to 0x280000400:2177) [ 9283.360869] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9296.892909] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9300.333295] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9312.592797] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 16:43:36 (1780346616) [ 9322.235552] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9328.148326] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9329.693138] Lustre: Failing over lustre-MDT0000 [ 9329.915599] Lustre: server umount lustre-MDT0000 complete [ 9349.739144] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9349.741801] LDISKFS-fs (dm-0): recovery complete [ 9349.747993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9363.038746] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9365.054617] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8897) [ 9365.058428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8929) [ 9368.993088] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9370.395259] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9373.984727] Lustre: Failing over lustre-MDT0001 [ 9374.226549] Lustre: server umount lustre-MDT0001 complete [ 9386.286112] LustreError: 226876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.38@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9386.319401] LustreError: 226876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 206 previous similar messages [ 9396.082547] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9396.085429] LDISKFS-fs (dm-1): recovery complete [ 9396.094254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9400.588521] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9401.904022] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2060 to 0x2c0000400:2209) [ 9401.909479] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2060 to 0x280000400:2209) [ 9407.651728] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9409.212982] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9416.506142] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 16:45:21 (1780346721) [ 9424.108914] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9430.260473] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9431.901024] Lustre: Failing over lustre-MDT0000 [ 9432.545174] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9432.554547] Lustre: Skipped 60 previous similar messages [ 9432.880974] Lustre: lustre-MDT0001: Client e3357f50-972d-425a-addd-7995c744c813 (at 192.168.204.38@tcp) reconnecting [ 9432.895587] Lustre: Skipped 1 previous similar message [ 9432.920251] Lustre: 226131:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8cad4132ca80 x1866819101017344/t68719476738(0) o36->e3357f50-972d-425a-addd-7995c744c813@192.168.204.38@tcp:125/0 lens 496/2888 e 0 to 0 dl 1780346750 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 9432.942083] Lustre: 226131:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 3 previous similar messages [ 9434.109385] Lustre: server umount lustre-MDT0000 complete [ 9437.090526] Lustre: Failing over lustre-MDT0001 [ 9437.091582] LustreError: 42147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346744 with bad export cookie 9786937278493166235 [ 9437.110537] LustreError: 42147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9437.425634] Lustre: server umount lustre-MDT0001 complete [ 9457.637860] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9457.639681] LDISKFS-fs (dm-0): recovery complete [ 9457.645900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9457.843688] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9457.846157] LDISKFS-fs (dm-1): recovery complete [ 9457.858088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.598283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9462.606609] Lustre: Skipped 15 previous similar messages [ 9462.651483] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9462.664708] Lustre: Skipped 15 previous similar messages [ 9463.906975] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9463.908835] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 9463.917207] Lustre: Skipped 16 previous similar messages [ 9463.928040] Lustre: Skipped 59 previous similar messages [ 9464.092916] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9464.104274] Lustre: Skipped 17 previous similar messages [ 9464.143074] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2060 to 0x280000400:2241) [ 9464.143424] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2060 to 0x2c0000400:2241) [ 9466.853092] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9467.003274] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9483.645868] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8483 to 0x2c0000401:8929) [ 9483.646505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8579 to 0x280000401:8961) [ 9487.085850] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9488.521157] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9489.878869] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9496.986640] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 16:46:42 (1780346802) [ 9498.548876] Lustre: 238014:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e3357f50-972d-425a-addd-7995c744c813 at adminstrative request [ 9505.530200] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 16:46:50 (1780346810) [ 9512.867148] Lustre: Failing over lustre-MDT0000 [ 9513.109572] Lustre: server umount lustre-MDT0000 complete [ 9531.397522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9540.581482] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x87d22f99f39d3d1c [ 9540.599765] LustreError: 238740:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGC192.168.204.138@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff8cae7cada500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9544.796666] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9546.381331] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8980 to 0x2c0000401:9025) [ 9546.388921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9013 to 0x280000401:9057) [ 9551.578722] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9553.167844] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9561.080320] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 16:47:46 (1780346866) [ 9577.076261] Lustre: Failing over lustre-OST0000 [ 9579.197780] Lustre: server umount lustre-OST0000 complete [ 9597.018331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9602.842077] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9609.865179] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9611.271751] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9620.952784] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 16:48:45 (1780346925) [ 9625.845202] Lustre: Failing over lustre-MDT0000 [ 9626.157668] Lustre: server umount lustre-MDT0000 complete [ 9632.969711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9637.122619] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9638.455610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9013 to 0x280000401:9089) [ 9638.462369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8980 to 0x2c0000401:9057) [ 9645.617114] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 16:49:10 (1780346950) [ 9653.008158] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9655.328262] Lustre: Failing over lustre-OST0000 [ 9655.452443] Lustre: server umount lustre-OST0000 complete [ 9676.512088] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9676.515540] LDISKFS-fs (dm-2): recovery complete [ 9676.526454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9682.766057] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9689.692594] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9691.021532] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9698.330057] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 16:50:03 (1780347003) [ 9705.224094] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9708.476380] Lustre: Failing over lustre-OST0000 [ 9708.542862] Lustre: server umount lustre-OST0000 complete [ 9710.049404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9710.055932] LustreError: Skipped 14 previous similar messages [ 9729.851965] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9729.855061] LDISKFS-fs (dm-2): recovery complete [ 9729.861984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9732.123038] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.38@tcp inode [0x2000301a1:0x5:0x0] object 0x280000401:9091 extent [0-4194303]: client csum 11645e7e, server csum dc4ab47c [ 9735.775315] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9742.147485] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9743.602756] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9750.813747] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 16:50:55 (1780347055) [ 9756.765417] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9762.585225] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9767.195097] Lustre: Failing over lustre-MDT0000 [ 9767.423420] Lustre: server umount lustre-MDT0000 complete [ 9769.971848] LustreError: 9479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780347076 with bad export cookie 9786937278493193514 [ 9769.988468] Lustre: Failing over lustre-OST0000 [ 9770.992835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9770.995811] Lustre: Skipped 8 previous similar messages [ 9772.110077] Lustre: server umount lustre-OST0000 complete [ 9790.850810] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9790.852752] LDISKFS-fs (dm-0): recovery complete [ 9790.857804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9797.296566] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9800.642666] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8980 to 0x2c0000401:9089) [ 9816.819461] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9816.822251] LDISKFS-fs (dm-2): recovery complete [ 9816.830580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9821.803281] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9834.535456] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 16:52:19 (1780347139) [ 9843.299326] Lustre: Failing over lustre-OST0000 [ 9843.430746] Lustre: server umount lustre-OST0000 complete [ 9846.202288] Lustre: Failing over lustre-MDT0000 [ 9846.454711] Lustre: server umount lustre-MDT0000 complete [ 9862.756424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9862.851531] LustreError: MGC192.168.204.138@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9862.862599] LustreError: Skipped 5 previous similar messages [ 9866.143854] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9868.305413] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8980 to 0x2c0000401:9121) [ 9872.184166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9876.122310] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9877.758130] Lustre: lustre-OST0000: Denying connection for new client 920bda6a-7fea-4de8-b260-7c91668e553a (at 192.168.204.38@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 9877.768826] Lustre: Skipped 2 previous similar messages [ 9882.925920] Lustre: lustre-OST0000: Denying connection for new client 920bda6a-7fea-4de8-b260-7c91668e553a (at 192.168.204.38@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 9888.041918] Lustre: lustre-OST0000: Denying connection for new client 920bda6a-7fea-4de8-b260-7c91668e553a (at 192.168.204.38@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 9898.281800] Lustre: lustre-OST0000: Denying connection for new client 920bda6a-7fea-4de8-b260-7c91668e553a (at 192.168.204.38@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:45 [ 9898.295566] Lustre: Skipped 1 previous similar message [ 9918.764827] Lustre: lustre-OST0000: Denying connection for new client 920bda6a-7fea-4de8-b260-7c91668e553a (at 192.168.204.38@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:24 [ 9918.783400] Lustre: Skipped 3 previous similar messages [ 9943.500214] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9943.509170] Lustre: 251059:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client f7c3117c-7c8a-4428-a99f-897403dd4289@ [ 9943.519443] Lustre: 251059:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 9943.525140] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9943.546639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9140 to 0x280000401:9161) [ 9945.862340] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 62 sec [ 9958.581518] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 9963.403586] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 16:54:28 (1780347268) [ 9966.281082] Lustre: Failing over lustre-OST0001 [ 9966.398132] Lustre: server umount lustre-OST0001 complete [ 9983.102874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9987.995389] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9995.367662] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 16:55:00 (1780347300) [ 9998.389103] Lustre: Failing over lustre-OST0000 [ 9998.485319] Lustre: server umount lustre-OST0000 complete [ 9998.823391] LustreError: 42146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9998.841023] LustreError: 42146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 173 previous similar messages [10014.229363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10015.853969] LustreError: 254220:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [10015.859627] LustreError: 254220:0:(ldlm_lib.c:2886:target_recovery_thread()) Skipped 76 previous similar messages [10018.484777] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10021.855466] Lustre: *** cfs_fail_loc=715, val=40*** [10032.095175] Lustre: 3666:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780347322/real 1780347322] req@ffff8cad4c7cf480 x1866819102803200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1780347338 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [10032.123199] Lustre: 3666:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [10032.131173] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [10032.140861] Lustre: Skipped 1 previous similar message [10038.239696] Lustre: *** cfs_fail_loc=715, val=40*** [10038.242405] Lustre: Skipped 2 previous similar messages [10047.787483] Lustre: lustre-OST0000: Client 920bda6a-7fea-4de8-b260-7c91668e553a (at 192.168.204.38@tcp) reconnected, waiting for 3 clients in recovery for 1:08 [10047.794766] Lustre: Skipped 1 previous similar message [10054.111136] Lustre: *** cfs_fail_loc=715, val=40*** [10054.113473] Lustre: Skipped 2 previous similar messages [10055.895395] LustreError: 254220:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 awake [10055.900363] LustreError: 254220:0:(ldlm_lib.c:2886:target_recovery_thread()) Skipped 75 previous similar messages [10058.617347] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10059.689987] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10065.364989] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 16:56:10 (1780347370) [10068.196020] Lustre: Failing over lustre-MDT0000 [10068.430935] Lustre: server umount lustre-MDT0000 complete [10068.965420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.983351] Lustre: Skipped 31 previous similar messages [10083.885925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.159574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10084.164857] Lustre: Skipped 12 previous similar messages [10084.197709] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10084.206147] Lustre: Skipped 12 previous similar messages [10086.904720] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10088.753954] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10088.761554] Lustre: Skipped 11 previous similar messages [10089.448515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10089.456197] Lustre: Skipped 34 previous similar messages [10089.463229] LustreError: 255655:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [10095.071319] Lustre: *** cfs_fail_loc=715, val=80*** [10095.078680] Lustre: Skipped 2 previous similar messages [10105.129448] Lustre: lustre-MDT0000: Client 920bda6a-7fea-4de8-b260-7c91668e553a (at 192.168.204.38@tcp) reconnected, waiting for 2 clients in recovery for 1:24 [10105.137962] Lustre: Skipped 2 previous similar messages [10105.823636] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10111.455412] Lustre: *** cfs_fail_loc=715, val=80*** [10111.459128] Lustre: Skipped 1 previous similar message [10121.184847] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10121.515422] Lustre: lustre-MDT0000: Client 920bda6a-7fea-4de8-b260-7c91668e553a (at 192.168.204.38@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [10127.327202] Lustre: *** cfs_fail_loc=715, val=80*** [10127.333872] Lustre: Skipped 1 previous similar message [10136.872136] Lustre: lustre-MDT0000: Client 920bda6a-7fea-4de8-b260-7c91668e553a (at 192.168.204.38@tcp) reconnected, waiting for 2 clients in recovery for 0:52 [10137.568127] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10143.711958] Lustre: *** cfs_fail_loc=715, val=80*** [10143.714236] Lustre: Skipped 2 previous similar messages [10153.257672] Lustre: lustre-MDT0000: Client 920bda6a-7fea-4de8-b260-7c91668e553a (at 192.168.204.38@tcp) reconnected, waiting for 2 clients in recovery for 0:36 [10153.955178] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10169.314750] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10169.487119] LustreError: 255655:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 awake [10169.504203] Lustre: 255655:0:(ldlm_lib.c:2932:target_recovery_thread()) too long recovery - read logs [10169.513733] LustreError: dumping log to /tmp/lustre-log.1780347476.255655 [10169.646162] Lustre: lustre-MDT0000: Recovery over after 1:21, of 2 clients 2 recovered and 0 were evicted. [10169.654027] Lustre: Skipped 12 previous similar messages [10169.676255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9134 to 0x2c0000401:9153) [10169.677159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9175 to 0x280000401:9193) [10172.566424] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10173.670962] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10180.146550] Lustre: DEBUG MARKER: == replay-single test complete, duration 9876 sec ======== 16:58:05 (1780347485) [10181.306319] Lustre: DEBUG MARKER: === replay-single: start cleanup 16:58:06 (1780347486) === [10189.888221] Lustre: DEBUG MARKER: === replay-single: finish cleanup 16:58:15 (1780347495) === [10226.014150] Lustre: server umount lustre-MDT0000 complete [10231.366922] LustreError: 167674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780347538 with bad export cookie 9786937278493206338 [10231.374228] LustreError: 167674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10231.575147] Lustre: server umount lustre-MDT0001 complete [10246.294979] Lustre: server umount lustre-OST0000 complete [10251.749974] Lustre: server umount lustre-OST0001 complete [10262.242488] Lustre: DEBUG MARKER: oleg438-server.virtnet: executing unload_modules_local [10264.089529] Key type lgssc unregistered [10264.307667] LNet: 258810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10264.312822] LNetError: 258810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10264.322728] LNet: Removed LNI 192.168.204.138@tcp [10264.855135] Key type .llcrypt unregistered [10264.859958] Key type ._llcrypt unregistered