[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492069907 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003310] x2apic enabled [ 0.004018] Switched APIC routing to physical x2apic. [ 0.005024] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009020] pid_max: default: 32768 minimum: 301 [ 0.011081] LSM: Security Framework initializing [ 0.012112] Yama: becoming mindful. [ 0.013080] SELinux: Initializing. [ 0.014150] *** VALIDATE selinux *** [ 0.023695] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028873] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029184] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030150] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032050] *** VALIDATE tmpfs *** [ 0.034053] *** VALIDATE proc *** [ 0.035380] *** VALIDATE cgroup *** [ 0.036019] *** VALIDATE cgroup2 *** [ 0.037330] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038186] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040020] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041043] Spectre V2 : User space: Vulnerable [ 0.042020] Speculative Store Bypass: Vulnerable [ 0.045793] debug: unmapping init [mem 0xffffffff8be59000-0xffffffff8be60fff] [ 0.048235] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049831] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050035] ... version: 2 [ 0.051022] ... bit width: 48 [ 0.052024] ... generic registers: 4 [ 0.053025] ... value mask: 0000ffffffffffff [ 0.054027] ... max period: 00007fffffffffff [ 0.055029] ... fixed-purpose events: 3 [ 0.056028] ... event mask: 000000070000000f [ 0.057411] rcu: Hierarchical SRCU implementation. [ 0.059899] smp: Bringing up secondary CPUs ... [ 0.060768] x86: Booting SMP configuration: [ 0.061050] .... node #0, CPUs: #1 #2 #3 [ 0.065148] smp: Brought up 1 node, 4 CPUs [ 0.067023] smpboot: Max logical packages: 1 [ 0.068028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.163721] node 0 deferred pages initialised in 93ms [ 0.167212] devtmpfs: initialized [ 0.168313] x86/mm: Memory block size: 128MB [ 0.171039] gcov: version magic: 0x41383552 [ 0.173318] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.177134] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.180391] pinctrl core: initialized pinctrl subsystem [ 0.182248] [ 0.182723] ************************************************************* [ 0.185023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187015] ** ** [ 0.190018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.192016] ** ** [ 0.194017] ** This means that this kernel is built to expose internal ** [ 0.197016] ** IOMMU data structures, which may compromise security on ** [ 0.199014] ** your system. ** [ 0.202020] ** ** [ 0.204015] ** If you see this message and you are not debugging the ** [ 0.207017] ** kernel, report this immediately to your vendor! ** [ 0.209015] ** ** [ 0.212017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.214016] ************************************************************* [ 0.217871] NET: Registered protocol family 16 [ 0.219501] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.222073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.225103] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.229046] cpuidle: using governor menu [ 0.231168] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.234688] PCI: Using configuration type 1 for base access [ 0.236156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.247143] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.250027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.254069] cryptd: max_cpu_qlen set to 1000 [ 0.256361] ACPI: Added _OSI(Module Device) [ 0.258023] ACPI: Added _OSI(Processor Device) [ 0.260019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.262018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.267753] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.276669] ACPI: Interpreter enabled [ 0.278075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.280028] ACPI: Using IOAPIC for interrupt routing [ 0.282183] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.286489] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.297479] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.299100] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.302027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.306109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.312410] acpiphp: Slot [2] registered [ 0.314141] acpiphp: Slot [5] registered [ 0.316185] acpiphp: Slot [6] registered [ 0.318148] acpiphp: Slot [7] registered [ 0.319147] acpiphp: Slot [8] registered [ 0.321187] acpiphp: Slot [9] registered [ 0.323147] acpiphp: Slot [10] registered [ 0.324134] acpiphp: Slot [3] registered [ 0.326163] acpiphp: Slot [4] registered [ 0.328185] acpiphp: Slot [11] registered [ 0.329147] acpiphp: Slot [12] registered [ 0.331175] acpiphp: Slot [13] registered [ 0.333138] acpiphp: Slot [14] registered [ 0.334375] acpiphp: Slot [15] registered [ 0.336156] acpiphp: Slot [16] registered [ 0.338167] acpiphp: Slot [17] registered [ 0.340148] acpiphp: Slot [18] registered [ 0.342170] acpiphp: Slot [19] registered [ 0.344186] acpiphp: Slot [20] registered [ 0.346105] acpiphp: Slot [21] registered [ 0.347132] acpiphp: Slot [22] registered [ 0.349156] acpiphp: Slot [23] registered [ 0.351202] acpiphp: Slot [24] registered [ 0.352109] acpiphp: Slot [25] registered [ 0.354312] acpiphp: Slot [26] registered [ 0.356146] acpiphp: Slot [27] registered [ 0.358116] acpiphp: Slot [28] registered [ 0.360136] acpiphp: Slot [29] registered [ 0.361124] acpiphp: Slot [30] registered [ 0.363179] acpiphp: Slot [31] registered [ 0.364056] PCI host bridge to bus 0000:00 [ 0.366023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.369026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.371023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.374026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.378040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.381032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.383176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.386328] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.390620] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.402812] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.407169] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.410030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.413032] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.416022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.418674] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.421926] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.426118] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.428774] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.434016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.448018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.454017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.461356] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.469025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.478023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.501023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.513078] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.526020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.535018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.561020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.571000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.586021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.602019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.635020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.650148] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.661022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.672023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.705020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.716366] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.728023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.736020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.755021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.764960] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.772019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.787020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.806021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.825530] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.828326] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.830317] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.832308] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.834215] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.839164] iommu: Default domain type: Passthrough [ 0.840444] SCSI subsystem initialized [ 0.841127] ACPI: bus type USB registered [ 0.843160] usbcore: registered new interface driver usbfs [ 0.844095] usbcore: registered new interface driver hub [ 0.846075] usbcore: registered new device driver usb [ 0.847145] pps_core: LinuxPPS API ver. 1 registered [ 0.849011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.851075] PTP clock support registered [ 0.853046] EDAC MC: Ver: 3.0.0 [ 0.854481] PCI: Using ACPI for IRQ routing [ 0.856401] NetLabel: Initializing [ 0.857017] NetLabel: domain hash size = 128 [ 0.858015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.859129] NetLabel: unlabeled traffic allowed by default [ 0.861152] vgaarb: loaded [ 0.862212] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.864012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.869367] clocksource: Switched to clocksource kvm-clock [ 0.979249] VFS: Disk quotas dquot_6.6.0 [ 0.981177] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.983913] *** VALIDATE ramfs *** [ 0.985195] *** VALIDATE hugetlbfs *** [ 0.986486] pnp: PnP ACPI init [ 0.988502] pnp: PnP ACPI: found 6 devices [ 1.018463] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.021522] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.023475] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.025297] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.027253] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.029276] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.032078] NET: Registered protocol family 2 [ 1.034138] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.039334] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.043692] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.050521] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.054902] TCP: Hash tables configured (established 65536 bind 65536) [ 1.059430] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.063576] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.066650] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.069916] NET: Registered protocol family 1 [ 1.073410] RPC: Registered named UNIX socket transport module. [ 1.075765] RPC: Registered udp transport module. [ 1.077566] RPC: Registered tcp transport module. [ 1.079456] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.081535] NET: Registered protocol family 44 [ 1.083394] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.085450] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.087706] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.090347] PCI: CLS 0 bytes, default 64 [ 1.092179] Unpacking initramfs... [ 2.533503] debug: unmapping init [mem 0xffff98773cc54000-0xffff98773ffbffff] [ 2.537854] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.540212] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.543162] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.050246] Initialise system trusted keyrings [ 3.052165] Key type blacklist registered [ 3.054697] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.065792] zbud: loaded [ 3.069338] *** VALIDATE nfs *** [ 3.070799] *** VALIDATE nfs4 *** [ 3.072654] pstore: using deflate compression [ 3.076533] Platform Keyring initialized [ 3.202277] NET: Registered protocol family 38 [ 3.204679] Key type asymmetric registered [ 3.206844] Asymmetric key parser 'x509' registered [ 3.209187] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.212497] io scheduler mq-deadline registered [ 3.214594] io scheduler kyber registered [ 3.216392] io scheduler bfq registered [ 3.218381] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.221535] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.224859] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.228165] ACPI: Power Button [PWRF] [ 3.233934] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.240930] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.255181] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.264334] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.282429] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.310473] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.338998] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.344429] Non-volatile memory driver v1.3 [ 3.346311] Linux agpgart interface v0.103 [ 3.384113] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.386482] vda: detected capacity change from 0 to 76681216 [ 3.405908] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.408987] vdb: detected capacity change from 0 to 1073741824 [ 3.431233] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.434975] vdc: detected capacity change from 0 to 2621440000 [ 3.457889] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.460790] vdd: detected capacity change from 0 to 2621440000 [ 3.488409] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.491427] vde: detected capacity change from 0 to 4294967296 [ 3.511187] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.514431] vdf: detected capacity change from 0 to 4294967296 [ 3.525228] libphy: Fixed MDIO Bus: probed [ 3.531343] usbcore: registered new interface driver usbserial_generic [ 3.534262] usbserial: USB Serial support registered for generic [ 3.537177] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.542251] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.544312] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.546822] mousedev: PS/2 mouse device common for all mice [ 3.549647] rtc_cmos 00:05: RTC can wake from S4 [ 3.552204] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.554303] rtc_cmos 00:05: registered as rtc0 [ 3.557920] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.560053] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.562484] intel_pstate: CPU model not supported [ 3.567939] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.570593] hid: raw HID events driver (C) Jiri Kosina [ 3.573467] usbcore: registered new interface driver usbhid [ 3.575918] usbhid: USB HID core driver [ 3.577567] drop_monitor: Initializing network drop monitor service [ 3.580456] Initializing XFRM netlink socket [ 3.582938] NET: Registered protocol family 10 [ 3.586276] Segment Routing with IPv6 [ 3.587467] NET: Registered protocol family 17 [ 3.589105] mpls_gso: MPLS GSO support [ 3.594956] RAS: Correctable Errors collector initialized. [ 3.596825] AVX version of gcm_enc/dec engaged. [ 3.598228] AES CTR mode by8 optimization enabled [ 3.673190] sched_clock: Marking stable (3673164454, 0)->(4733918008, -1060753554) [ 3.675987] registered taskstats version 1 [ 3.677590] Loading compiled-in X.509 certificates [ 3.679239] zswap: loaded using pool lzo/zbud [ 3.702976] Key type big_key registered [ 3.714865] Key type encrypted registered [ 3.716438] ima: No TPM chip found, activating TPM-bypass! [ 3.718552] ima: Allocated hash algorithm: sha1 [ 3.720787] ima: No architecture policies found [ 3.722798] evm: Initialising EVM extended attributes: [ 3.724654] evm: security.selinux [ 3.725965] evm: security.ima [ 3.727108] evm: security.capability [ 3.728582] evm: HMAC attrs: 0x1 [ 3.731183] rtc_cmos 00:05: setting system clock to 2026-09-06 09:37:40 UTC (1788687460) [ 3.737453] debug: unmapping init [mem 0xffffffff8ce03000-0xffffffff8cffffff] [ 3.740586] debug: unmapping init [mem 0xffffffff8bb82000-0xffffffff8be58fff] [ 3.749109] Write protecting the kernel read-only data: 28672k [ 3.752804] debug: unmapping init [mem 0xffffffff8a203000-0xffffffff8a3fffff] [ 3.755835] debug: unmapping init [mem 0xffffffff8ab14000-0xffffffff8abfffff] [ 3.795156] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.803405] systemd[1]: Detected virtualization kvm. [ 3.805790] systemd[1]: Detected architecture x86-64. [ 3.808155] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.838165] systemd[1]: No hostname configured. [ 3.840114] systemd[1]: Set hostname to . [ 3.842501] random: systemd: uninitialized urandom read (16 bytes read) [ 3.845200] systemd[1]: Initializing machine ID from random generator. [ 3.898306] random: ln: uninitialized urandom read (6 bytes read) [ 3.997837] random: systemd: uninitialized urandom read (16 bytes read) [ 4.000573] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.004681] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.009024] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.663674] device-mapper: uevent: version 1.0.3 [ 4.666606] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.511483] random: fast init done [ 5.530432] virtio_net virtio0 ens2: renamed from eth0 [ 5.555798] scsi host0: ata_piix [ 5.589739] scsi host1: ata_piix [ 5.591281] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.594495] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.216567] dracut-initqueue[593]: RTNETLINK answers: File exists [ 10.239825] random: crng init done [ 10.241575] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.815369] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.030873] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.298469] SELinux: Disabled at runtime. [ 12.367378] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.379567] systemd[1]: Detected virtualization kvm. [ 12.381124] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.901849] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.905911] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.910966] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.916857] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.920977] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.928319] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.936895] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ 13.121949] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.380385] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.772112] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.782834] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.892768] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.907382] EDAC sbridge: Ver: 1.1.2 [ 15.901862] Key type dns_resolver registered [ 16.230353] NFS: Registering the id_resolver key type [ 16.232439] Key type id_resolver registered [ 16.234425] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg640-server login: [ 74.445941] libcfs: loading out-of-tree module taints kernel. [ 74.490937] Key type ._llcrypt registered [ 74.492705] Key type .llcrypt registered [ 74.597547] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 92.785803] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 95.153721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 95.185629] alg: No test for adler32 (adler32-zlib) [ 97.101076] Lustre: Lustre: Build Version: 2.17.58_2_g2f2534e [ 98.276238] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 100.033410] Key type lgssc registered [ 102.253678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.382427] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.832503] hrtimer: interrupt took 11385499 ns [ 162.018393] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 179.391220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 179.530913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.021368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 181.058389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 181.162693] Lustre: lustre-MDT0000: new disk, initializing [ 181.269735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.283479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 186.009909] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.889798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.054974] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 199.108711] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 199.114061] Lustre: Skipped 1 previous similar message [ 199.196947] Lustre: lustre-MDT0001: new disk, initializing [ 199.269068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.299396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 199.310047] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.854760] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.740354] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 217.209337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.478292] Lustre: lustre-OST0000: new disk, initializing [ 217.481458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 217.489496] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 217.540769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.227755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 222.234528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 222.326784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.962739] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.729067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.950929] Lustre: lustre-OST0001: new disk, initializing [ 235.961576] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 235.969853] Lustre: 9521:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 236.040567] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.196737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 241.202765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 241.247843] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 241.645522] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.467808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.375847] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 270.812915] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing check_logdir /tmp/testlogs/ [ 275.459598] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing yml_node [ 279.549590] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 281.964362] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 284.174771] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 285.578602] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sun Sep 6 05:42:20 EDT 2026 [ 299.235847] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 300.971363] Lustre: DEBUG MARKER: === replay-single: start setup 05:42:36 (1788687756) === [ 305.941634] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing check_config_client /mnt/lustre [ 323.006606] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 325.960222] Lustre: 13327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 329.216669] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 333.913982] Lustre: DEBUG MARKER: === replay-single: finish setup 05:43:09 (1788687789) === [ 335.518477] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 05:43:10 (1788687790) [ 342.886677] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 344.792022] Lustre: Failing over lustre-MDT0000 [ 345.140491] Lustre: server umount lustre-MDT0000 complete [ 347.615710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 347.624150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.643416] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 348.671130] Lustre: Skipped 1 previous similar message [ 351.492979] LustreError: 13863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 351.511309] LustreError: 13863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 353.763038] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.797788] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 356.610710] LustreError: 13863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 358.885378] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 358.906661] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 364.012173] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.036574] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 364.495100] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788687805/real 1788687805] req@ffff987687b16a00 x1875574841778944/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788687821 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 364.547516] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 368.088059] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 368.093831] LDISKFS-fs (dm-0): recovery complete [ 368.137907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 372.991321] LustreError: 8747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.029117] LustreError: 8747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 374.753957] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877b7638380 x1875574841785728/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 375.116442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.770834] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 379.757610] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 380.392063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 380.453910] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 389.421155] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 391.104523] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 400.915499] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 05:44:15 (1788687855) [ 403.235447] Lustre: Failing over lustre-OST0000 [ 403.498608] Lustre: server umount lustre-OST0000 complete [ 405.991458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 406.021980] Lustre: Skipped 1 previous similar message [ 406.036414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 406.045786] LustreError: 8752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 406.066720] LustreError: 8752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 423.277633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 423.579915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.991952] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 425.147484] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 425.158849] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 425.169167] Lustre: Skipped 3 previous similar messages [ 430.994942] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 441.050520] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 443.343555] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 452.924525] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 05:45:07 (1788687907) [ 460.285993] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 462.452381] Lustre: Failing over lustre-MDT0000 [ 462.697157] Lustre: server umount lustre-MDT0000 complete [ 464.864702] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 464.867558] LustreError: 13863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 464.881698] Lustre: Skipped 3 previous similar messages [ 464.894693] LustreError: 13863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 480.535556] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788687921/real 1788687921] req@ffff9877bfa88700 x1875574841844736/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788687937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 480.575356] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 484.948802] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 484.951720] LDISKFS-fs (dm-0): recovery complete [ 484.957739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.193318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.201855] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 496.609678] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 496.612079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 496.634256] Lustre: Skipped 1 previous similar message [ 498.413386] Lustre: lustre-MDT0000: Denying connection for new client 8f82f27d-932c-4e93-97ff-c48b111074c8 (at 192.168.206.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 503.552691] Lustre: lustre-MDT0000: Denying connection for new client 8f82f27d-932c-4e93-97ff-c48b111074c8 (at 192.168.206.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 508.674892] Lustre: lustre-MDT0000: Denying connection for new client 8f82f27d-932c-4e93-97ff-c48b111074c8 (at 192.168.206.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 513.792589] Lustre: lustre-MDT0000: Denying connection for new client 8f82f27d-932c-4e93-97ff-c48b111074c8 (at 192.168.206.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 518.911430] Lustre: lustre-MDT0000: Denying connection for new client 8f82f27d-932c-4e93-97ff-c48b111074c8 (at 192.168.206.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:47 [ 529.151753] Lustre: lustre-MDT0000: Denying connection for new client 8f82f27d-932c-4e93-97ff-c48b111074c8 (at 192.168.206.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:37 [ 529.178450] Lustre: Skipped 1 previous similar message [ 549.638353] Lustre: lustre-MDT0000: Denying connection for new client 8f82f27d-932c-4e93-97ff-c48b111074c8 (at 192.168.206.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:16 [ 549.665453] Lustre: Skipped 3 previous similar messages [ 566.500142] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 566.503801] Lustre: 18230:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 4b06a278-f10a-497c-92c8-a8072deecfe6@ [ 566.519390] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 566.537876] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 566.538194] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 566.549269] Lustre: Skipped 2 previous similar messages [ 566.573237] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 566.580281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 579.774037] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 05:47:14 (1788688034) [ 587.150436] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 589.436303] Lustre: Failing over lustre-MDT0000 [ 589.722108] Lustre: server umount lustre-MDT0000 complete [ 592.351612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 592.358652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.375886] Lustre: Skipped 1 previous similar message [ 592.393701] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 592.423125] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [ 610.783325] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688050/real 1788688050] req@ffff9877b4b0df80 x1875574841906560/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788688066 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 610.808909] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.082612] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 613.086171] LDISKFS-fs (dm-0): recovery complete [ 613.095829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.546280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 621.605409] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 626.660896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 626.665618] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 626.909865] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 629.222827] Lustre: lustre-MDT0000: Denying connection for new client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 629.238666] Lustre: Skipped 3 previous similar messages [ 696.068797] Lustre: lustre-MDT0000: Denying connection for new client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:00 [ 696.084254] Lustre: Skipped 12 previous similar messages [ 696.500384] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 696.505077] Lustre: 19960:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 8f82f27d-932c-4e93-97ff-c48b111074c8@ [ 696.527532] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 696.545682] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 696.548641] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 696.573523] Lustre: Skipped 2 previous similar messages [ 696.619129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 696.620102] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 710.166725] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 05:49:25 (1788688165) [ 718.845844] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 721.270515] Lustre: Failing over lustre-MDT0000 [ 721.501605] Lustre: server umount lustre-MDT0000 complete [ 721.703578] LustreError: 8747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.718383] LustreError: 8747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [ 722.406183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 722.428327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 722.450978] Lustre: Skipped 3 previous similar messages [ 739.792102] Lustre: 3645:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688180/real 1788688180] req@ffff9876876c7800 x1875574841966080/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788688196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 739.831712] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 745.543296] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 745.549222] LDISKFS-fs (dm-0): recovery complete [ 745.570195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.050504] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877bf814700 x1875574841974528/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 750.330367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.396899] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 751.457150] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 754.758706] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 755.684223] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 755.757961] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 755.794372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 755.794390] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 765.118332] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 766.763740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 776.140799] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 05:50:31 (1788688231) [ 783.485444] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 785.208371] Lustre: Failing over lustre-MDT0000 [ 785.448797] Lustre: server umount lustre-MDT0000 complete [ 786.400431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 786.403464] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.420133] Lustre: Skipped 5 previous similar messages [ 802.799391] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688243/real 1788688243] req@ffff987784078e00 x1875574842000256/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788688259 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 802.846519] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 807.226051] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 807.229237] LDISKFS-fs (dm-0): recovery complete [ 807.237678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 813.029148] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877b57f4000 x1875574842009728/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 813.347242] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 814.858671] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 818.596975] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 818.671507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 818.682817] Lustre: Skipped 3 previous similar messages [ 818.830984] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 818.909274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 818.910257] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 830.299957] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 832.276347] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 842.285345] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 05:51:37 (1788688297) [ 850.414095] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 852.575586] Lustre: Failing over lustre-MDT0000 [ 854.498856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 854.523501] Lustre: Skipped 3 previous similar messages [ 854.824156] Lustre: server umount lustre-MDT0000 complete [ 870.879428] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688311/real 1788688311] req@ffff9877b76be300 x1875574842038016/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788688327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 870.904993] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 877.184754] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 877.188347] LDISKFS-fs (dm-0): recovery complete [ 877.199786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.120186] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff987784078000 x1875574842046464/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 881.375983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.387499] Lustre: Skipped 1 previous similar message [ 881.436369] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 882.513295] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 885.869686] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 886.756785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 886.764739] Lustre: Skipped 3 previous similar messages [ 886.968261] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 887.034287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 887.034634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:193) [ 896.327849] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 897.993735] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 906.673467] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 05:52:41 (1788688361) [ 915.993398] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 918.176686] Lustre: Failing over lustre-MDT0000 [ 918.307760] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 918.457459] Lustre: server umount lustre-MDT0000 complete [ 920.544732] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.564360] Lustre: Skipped 3 previous similar messages [ 936.927354] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688377/real 1788688377] req@ffff9877af0ab480 x1875574842075136/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788688393 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 936.952873] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.167624] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 940.169818] LDISKFS-fs (dm-0): recovery complete [ 940.182614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.186550] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8d562f1 [ 947.449701] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 950.024757] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 951.280751] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 952.940881] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 952.996331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 952.998342] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 961.444481] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 962.803970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 971.739645] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 05:53:46 (1788688426) [ 980.973946] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 983.302547] Lustre: Failing over lustre-MDT0000 [ 983.526705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 983.535021] Lustre: Skipped 1 previous similar message [ 983.685135] Lustre: server umount lustre-MDT0000 complete [ 985.891699] LustreError: 13863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.918670] LustreError: 13863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 134 previous similar messages [ 1005.027066] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688445/real 1788688445] req@ffff9876876c4000 x1875574842113280/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788688461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1005.068699] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1005.852271] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1005.859759] LDISKFS-fs (dm-0): recovery complete [ 1005.873699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1014.263042] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8d5683f [ 1014.543423] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1015.729046] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1019.028863] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1019.900743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1019.906997] Lustre: Skipped 9 previous similar messages [ 1020.000420] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1020.041452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1020.041678] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1028.446244] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1030.216262] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1039.282678] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 05:54:54 (1788688494) [ 1040.414909] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1040.421644] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1040.430439] LustreError: 6520:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877af226d80 x1875574820852352/t38654705666(0) o35->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:643/0 lens 392/456 e 0 to 0 dl 1788688508 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1048.502095] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1050.501954] Lustre: Failing over lustre-MDT0000 [ 1050.598430] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.601838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1050.612824] Lustre: Skipped 7 previous similar messages [ 1050.623781] Lustre: Skipped 4 previous similar messages [ 1051.003736] Lustre: server umount lustre-MDT0000 complete [ 1072.099046] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688512/real 1788688512] req@ffff987784079f80 x1875574842150400/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788688528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1072.131218] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1074.368317] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1074.372582] LDISKFS-fs (dm-0): recovery complete [ 1074.383746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.317788] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8d56d2b [ 1081.658040] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1086.379744] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1087.067801] Lustre: 6520:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877be51b100 x1875574820852352/t38654705666(0) o35->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:689/0 lens 392/456 e 0 to 0 dl 1788688554 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1087.115569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1087.127619] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1096.914164] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1099.168855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1109.345608] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 05:56:04 (1788688564) [ 1117.704987] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1119.629941] Lustre: Failing over lustre-MDT0000 [ 1119.984523] Lustre: server umount lustre-MDT0000 complete [ 1122.783819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1144.557418] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1144.562080] LDISKFS-fs (dm-0): recovery complete [ 1144.581616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.742624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.750859] Lustre: Skipped 3 previous similar messages [ 1148.815404] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1150.485174] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1150.509352] Lustre: Skipped 1 previous similar message [ 1153.206106] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1154.116744] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1154.141991] Lustre: Skipped 1 previous similar message [ 1154.205717] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1154.206179] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1162.785526] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1164.458711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1172.841176] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 05:57:08 (1788688628) [ 1179.999374] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1180.856169] Lustre: *** cfs_fail_loc=114, val=0*** [ 1183.824837] Lustre: Failing over lustre-MDT0000 [ 1183.981473] Lustre: server umount lustre-MDT0000 complete [ 1200.991360] Lustre: 3645:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688641/real 1788688641] req@ffff9877b48c1c00 x1875574842223488/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788688657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1201.027545] Lustre: 3645:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1201.033463] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1201.043422] LustreError: Skipped 1 previous similar message [ 1205.915531] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1205.918797] LDISKFS-fs (dm-0): recovery complete [ 1205.937571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.639613] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1216.071406] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1216.073226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1226.142832] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1227.851432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1237.751852] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 05:58:12 (1788688692) [ 1247.242269] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1248.312525] Lustre: *** cfs_fail_loc=128, val=0*** [ 1251.099181] Lustre: Failing over lustre-MDT0000 [ 1251.747617] Lustre: server umount lustre-MDT0000 complete [ 1251.819642] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1276.742649] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1276.745889] LDISKFS-fs (dm-0): recovery complete [ 1276.766685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.528568] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877af224e00 x1875574842270720/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1283.023595] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1283.031381] Lustre: Skipped 1 previous similar message [ 1287.826562] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1288.184277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1288.199017] Lustre: Skipped 16 previous similar messages [ 1288.285200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1288.286290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1297.357791] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1298.904890] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1307.194237] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 05:59:22 (1788688762) [ 1313.896515] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1316.139750] Lustre: Failing over lustre-MDT0000 [ 1316.415633] Lustre: server umount lustre-MDT0000 complete [ 1318.883806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1318.892357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.908883] Lustre: Skipped 16 previous similar messages [ 1339.612244] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1339.615028] LDISKFS-fs (dm-0): recovery complete [ 1339.620792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.880703] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1350.497766] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1350.502539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1361.034366] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1362.635125] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1372.949425] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 06:00:28 (1788688828) [ 1383.041088] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1385.403219] Lustre: Failing over lustre-MDT0000 [ 1385.593771] Lustre: server umount lustre-MDT0000 complete [ 1411.380866] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1411.386465] LDISKFS-fs (dm-0): recovery complete [ 1411.401392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.961862] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1416.976199] Lustre: Skipped 1 previous similar message [ 1417.473505] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1417.496907] Lustre: Skipped 3 previous similar messages [ 1421.816865] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1422.493371] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1422.507721] Lustre: Skipped 3 previous similar messages [ 1422.549162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1422.549839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1432.469630] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1434.898432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1444.251757] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 06:01:39 (1788688899) [ 1451.121568] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1459.041185] Lustre: Failing over lustre-MDT0000 [ 1459.463756] Lustre: server umount lustre-MDT0000 complete [ 1479.880097] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1479.888466] LDISKFS-fs (dm-0): recovery complete [ 1479.899611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.056440] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.072776] LustreError: Skipped 3 previous similar messages [ 1484.872660] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1488.612877] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1488.617685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1493.975610] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1495.589486] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1520.517865] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 06:02:56 (1788688976) [ 1529.389272] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1531.487934] Lustre: Failing over lustre-MDT0000 [ 1531.864857] Lustre: server umount lustre-MDT0000 complete [ 1534.943921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1534.963878] LustreError: 7982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.983734] LustreError: 7982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 259 previous similar messages [ 1552.863324] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788688993/real 1788688993] req@ffff9877846bbb80 x1875574842479488/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788689009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1552.886401] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1553.740532] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1553.742586] LDISKFS-fs (dm-0): recovery complete [ 1553.746701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.106937] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877844b6680 x1875574842489216/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1568.522981] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1568.947926] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1568.952390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1578.189563] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1580.015963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1592.427944] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 06:04:07 (1788689047) [ 1600.499672] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1602.280201] Lustre: Failing over lustre-MDT0000 [ 1602.481474] Lustre: server umount lustre-MDT0000 complete [ 1625.000345] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1625.005596] LDISKFS-fs (dm-0): recovery complete [ 1625.024211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1630.688041] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877b484ce00 x1875574842531840/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1636.070243] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1636.484711] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1636.490449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1645.833342] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1647.659240] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1655.611462] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 06:05:10 (1788689110) [ 1663.679735] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1665.661492] Lustre: Failing over lustre-MDT0000 [ 1667.902351] Lustre: server umount lustre-MDT0000 complete [ 1691.936805] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1691.939408] LDISKFS-fs (dm-0): recovery complete [ 1691.949146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.644520] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8d6332a [ 1692.928559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.930890] Lustre: Skipped 7 previous similar messages [ 1692.978542] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1692.992710] Lustre: Skipped 3 previous similar messages [ 1697.776103] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1698.463265] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1698.467373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1707.486372] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1708.935963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1717.647715] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 06:06:12 (1788689172) [ 1724.748284] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1726.816764] Lustre: Failing over lustre-MDT0000 [ 1727.053530] Lustre: server umount lustre-MDT0000 complete [ 1748.489427] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1748.498373] LDISKFS-fs (dm-0): recovery complete [ 1748.521952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.594194] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877be5bfb80 x1875574842602880/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1758.925551] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1760.408847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1760.410251] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1768.499261] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1770.194767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1777.750747] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 06:07:13 (1788689233) [ 1785.094711] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1787.056190] Lustre: Failing over lustre-MDT0000 [ 1787.264866] Lustre: server umount lustre-MDT0000 complete [ 1809.305001] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1809.307511] LDISKFS-fs (dm-0): recovery complete [ 1809.315706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.648612] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1822.697359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1822.710370] Lustre: Skipped 32 previous similar messages [ 1822.845268] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1822.847143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1830.896942] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1832.552517] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1841.604620] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 06:08:16 (1788689296) [ 1849.878537] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1852.101272] Lustre: Failing over lustre-MDT0000 [ 1852.503740] Lustre: server umount lustre-MDT0000 complete [ 1853.409464] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.436326] Lustre: Skipped 30 previous similar messages [ 1876.231476] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1876.233281] LDISKFS-fs (dm-0): recovery complete [ 1876.240226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.039159] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8d64368 [ 1884.379626] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1885.778637] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1885.782178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1893.645237] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1895.542716] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1905.290561] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 06:09:20 (1788689360) [ 1912.691685] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1914.911019] Lustre: Failing over lustre-MDT0000 [ 1915.195824] Lustre: server umount lustre-MDT0000 complete [ 1936.540982] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1936.546370] LDISKFS-fs (dm-0): recovery complete [ 1936.555562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.299849] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1943.313696] Lustre: Skipped 7 previous similar messages [ 1948.174525] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1948.764459] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1948.773558] Lustre: Skipped 7 previous similar messages [ 1948.811773] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1948.813701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1958.029507] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1960.058200] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1969.049775] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 06:10:24 (1788689424) [ 1977.033417] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1978.910336] Lustre: Failing over lustre-MDT0000 [ 1979.224944] Lustre: server umount lustre-MDT0000 complete [ 1996.255193] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1996.270122] LustreError: Skipped 7 previous similar messages [ 2002.604347] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2002.607934] LDISKFS-fs (dm-0): recovery complete [ 2002.615280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2006.499443] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877910f3100 x1875574842748928/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2006.516041] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 20 previous similar messages [ 2011.677514] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2012.201800] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 2012.201935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 2021.357556] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2022.783911] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2031.354274] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 06:11:26 (1788689486) [ 2039.547929] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2041.409679] Lustre: Failing over lustre-MDT0000 [ 2043.673258] Lustre: server umount lustre-MDT0000 complete [ 2066.415067] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2066.417132] LDISKFS-fs (dm-0): recovery complete [ 2066.423777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.577958] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2074.159870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 2074.169209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 2081.638719] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2083.438288] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2091.463722] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 06:12:26 (1788689546) [ 2097.641577] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2099.454644] Lustre: Failing over lustre-MDT0000 [ 2099.689620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2099.784421] Lustre: server umount lustre-MDT0000 complete [ 2121.187480] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788689561/real 1788689561] req@ffff9877b4722d80 x1875574842810496/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788689577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2121.233790] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2123.393295] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2123.395222] LDISKFS-fs (dm-0): recovery complete [ 2123.403221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.871597] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2137.150698] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 2137.152587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 2144.494953] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2145.955045] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2153.546645] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 06:13:28 (1788689608) [ 2161.331485] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2163.263867] Lustre: Failing over lustre-MDT0000 [ 2163.481570] LustreError: 13863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2163.500657] LustreError: 13863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 334 previous similar messages [ 2163.572293] Lustre: server umount lustre-MDT0000 complete [ 2186.167200] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2186.173375] LDISKFS-fs (dm-0): recovery complete [ 2186.192352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.893112] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8d65f84 [ 2199.701076] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2199.701535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2199.755111] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2207.815181] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2209.129126] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2217.117414] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 06:14:32 (1788689672) [ 2223.814467] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2225.855913] Lustre: Failing over lustre-MDT0000 [ 2225.933809] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 2226.055439] Lustre: server umount lustre-MDT0000 complete [ 2247.098631] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2247.100795] LDISKFS-fs (dm-0): recovery complete [ 2247.107566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2256.876985] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877af1f6a00 x1875574842892032/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2256.896618] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 2257.205649] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2257.221216] Lustre: Skipped 8 previous similar messages [ 2261.689528] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2262.614597] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 2262.623725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 2271.190289] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2272.667689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2281.317894] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 06:15:36 (1788689736) [ 2288.303651] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2290.263146] Lustre: Failing over lustre-MDT0000 [ 2290.567473] Lustre: server umount lustre-MDT0000 complete [ 2312.186663] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2312.189541] LDISKFS-fs (dm-0): recovery complete [ 2312.195923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.112266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.123183] Lustre: Skipped 9 previous similar messages [ 2323.880981] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2324.582490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:993) [ 2324.582974] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 2333.619796] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2335.213883] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2343.637412] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 06:16:38 (1788689798) [ 2350.972292] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2353.197911] Lustre: Failing over lustre-MDT0000 [ 2353.596782] Lustre: server umount lustre-MDT0000 complete [ 2376.867822] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2376.871694] LDISKFS-fs (dm-0): recovery complete [ 2376.885685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.580813] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2386.104080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2386.119552] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2395.033260] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2396.609576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2404.967788] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 06:17:40 (1788689860) [ 2411.333526] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2413.442274] Lustre: Failing over lustre-MDT0000 [ 2413.767994] Lustre: server umount lustre-MDT0000 complete [ 2435.292818] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2435.295748] LDISKFS-fs (dm-0): recovery complete [ 2435.302643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.460628] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2448.880857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2448.887955] Lustre: Skipped 41 previous similar messages [ 2448.995065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2448.995522] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2458.084058] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2459.760294] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2468.359090] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 06:18:43 (1788689923) [ 2476.412708] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2479.115144] Lustre: Failing over lustre-MDT0000 [ 2479.550276] Lustre: server umount lustre-MDT0000 complete [ 2479.587404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2479.597690] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.599760] LustreError: Skipped 3 previous similar messages [ 2479.636467] Lustre: Skipped 40 previous similar messages [ 2502.953705] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2502.957688] LDISKFS-fs (dm-0): recovery complete [ 2502.974335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.313719] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8d67c3a [ 2515.042671] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2516.064883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2516.065438] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2524.161874] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2525.539867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2533.136554] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 06:19:48 (1788689988) [ 2537.766890] Lustre: 75349:0:(genops.c:1774:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c706ebfb-b014-459a-b99d-0af5bebd3a85 at adminstrative request [ 2543.951477] Lustre: Failing over lustre-MDT0000 [ 2544.251667] Lustre: server umount lustre-MDT0000 complete [ 2562.230580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2562.815027] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2562.827128] Lustre: Skipped 9 previous similar messages [ 2566.136544] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2567.676863] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2567.685622] Lustre: Skipped 9 previous similar messages [ 2567.745439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2567.746334] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2575.136035] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2576.881706] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2583.266430] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2592.199986] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2598.263914] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 06:20:53 (1788690053) [ 2599.181710] Lustre: 77373:0:(genops.c:1774:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c706ebfb-b014-459a-b99d-0af5bebd3a85 at adminstrative request [ 2608.826583] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 06:21:04 (1788690064) [ 2615.514805] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2617.591340] Lustre: Failing over lustre-MDT0000 [ 2617.757962] Lustre: server umount lustre-MDT0000 complete [ 2635.231122] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2635.251086] LustreError: Skipped 9 previous similar messages [ 2642.570496] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2642.573236] LDISKFS-fs (dm-0): recovery complete [ 2642.586705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.476788] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877af156a00 x1875574843117056/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2645.524691] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 2651.257588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 2651.258067] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 2651.893833] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2662.053446] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2663.978611] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2672.418742] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 06:22:07 (1788690127) [ 2679.190410] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2680.974139] Lustre: Failing over lustre-MDT0000 [ 2681.233476] Lustre: server umount lustre-MDT0000 complete [ 2702.099219] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2702.100771] LDISKFS-fs (dm-0): recovery complete [ 2702.105896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2707.959474] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8d68c9b [ 2712.224588] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2713.675253] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2713.677406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2721.400461] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2723.069332] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2732.006957] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 06:23:07 (1788690187) [ 2738.727432] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2740.729239] Lustre: Failing over lustre-MDT0000 [ 2741.006123] Lustre: server umount lustre-MDT0000 complete [ 2761.183887] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788690201/real 1788690201] req@ffff9877be590e00 x1875574843181184/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788690217 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2761.219017] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2763.081572] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2763.084184] LDISKFS-fs (dm-0): recovery complete [ 2763.091493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2764.551234] LustreError: 56890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2764.572216] LustreError: 56890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 308 previous similar messages [ 2775.733885] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2777.190396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2777.198566] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2784.961184] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2786.865515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2795.378908] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 06:24:10 (1788690250) [ 2803.017710] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2804.637388] Lustre: Failing over lustre-MDT0000 [ 2804.848990] Lustre: server umount lustre-MDT0000 complete [ 2825.474621] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2825.478590] LDISKFS-fs (dm-0): recovery complete [ 2825.490746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.818794] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2839.112607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2839.112868] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2847.853651] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2849.404871] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2857.295741] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 06:25:12 (1788690312) [ 2864.240180] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2866.026720] Lustre: Failing over lustre-MDT0000 [ 2866.344444] Lustre: server umount lustre-MDT0000 complete [ 2887.936560] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2887.939524] LDISKFS-fs (dm-0): recovery complete [ 2887.946402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.705318] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2895.712595] Lustre: Skipped 9 previous similar messages [ 2900.099358] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2901.048333] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 2901.049960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 2910.290698] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2912.439582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2922.283152] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 06:26:17 (1788690377) [ 2930.653399] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2932.990325] Lustre: Failing over lustre-MDT0000 [ 2933.426655] Lustre: server umount lustre-MDT0000 complete [ 2957.966827] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2957.974084] LDISKFS-fs (dm-0): recovery complete [ 2957.986930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2962.406484] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8d6a289 [ 2962.766984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2962.776843] Lustre: Skipped 9 previous similar messages [ 2968.013612] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2968.218029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2968.233268] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2977.500795] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2979.087896] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2987.883276] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 06:27:23 (1788690443) [ 2994.876525] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2996.895489] Lustre: Failing over lustre-MDT0000 [ 2997.149096] Lustre: server umount lustre-MDT0000 complete [ 3020.922743] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3020.926019] LDISKFS-fs (dm-0): recovery complete [ 3020.937331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.393142] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3030.145344] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 3030.146598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 3038.470909] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3040.102378] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3048.832301] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 06:28:24 (1788690504) [ 3056.150719] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3058.128802] Lustre: Failing over lustre-MDT0000 [ 3058.324311] Lustre: server umount lustre-MDT0000 complete [ 3079.908630] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3079.913290] LDISKFS-fs (dm-0): recovery complete [ 3079.930817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.818930] Lustre: MGC192.168.206.140@tcp: Connection restored to 0@lo (at 0@lo) [ 3086.824304] Lustre: Skipped 43 previous similar messages [ 3091.153490] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3092.560296] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 3092.563827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 3100.966286] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3102.528280] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3110.423387] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 06:29:25 (1788690565) [ 3117.537080] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3119.601710] Lustre: Failing over lustre-MDT0000 [ 3119.913101] Lustre: server umount lustre-MDT0000 complete [ 3123.175867] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.178440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3123.190305] Lustre: Skipped 39 previous similar messages [ 3123.208380] LustreError: Skipped 6 previous similar messages [ 3142.080810] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3142.082628] LDISKFS-fs (dm-0): recovery complete [ 3142.098452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3152.959144] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3154.015188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3154.019606] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3161.018516] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3162.558801] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3170.015860] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 06:30:25 (1788690625) [ 3177.314800] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3179.042465] Lustre: Failing over lustre-MDT0000 [ 3179.301314] Lustre: server umount lustre-MDT0000 complete [ 3201.504253] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3201.508235] LDISKFS-fs (dm-0): recovery complete [ 3201.526689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3211.287574] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8d6ba92 [ 3211.294540] Lustre: Skipped 2 previous similar messages [ 3212.756486] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3212.762156] Lustre: Skipped 9 previous similar messages [ 3215.474404] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3216.918415] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3216.931064] Lustre: Skipped 9 previous similar messages [ 3216.979788] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 3216.980029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3224.739699] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3226.285343] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3234.689121] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 06:31:29 (1788690689) [ 3242.066888] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3243.821694] Lustre: Failing over lustre-MDT0000 [ 3244.112647] Lustre: server umount lustre-MDT0000 complete [ 3263.968960] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3263.992394] LustreError: Skipped 9 previous similar messages [ 3265.202803] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3265.213527] LDISKFS-fs (dm-0): recovery complete [ 3265.227713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.207584] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98768a5bf480 x1875574843484544/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3274.236535] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 11 previous similar messages [ 3279.100316] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3279.989508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 3279.992199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 3289.545482] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3291.054126] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3299.941578] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 06:32:35 (1788690755) [ 3301.046040] Lustre: 98967:0:(genops.c:1774:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c706ebfb-b014-459a-b99d-0af5bebd3a85 at adminstrative request [ 3309.696641] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 06:32:44 (1788690764) [ 3316.730857] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3318.547695] Lustre: Failing over lustre-MDT0000 [ 3318.725880] Lustre: server umount lustre-MDT0000 complete [ 3333.029630] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3333.033336] LDISKFS-fs (dm-0): recovery complete [ 3333.042989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.323118] Lustre: lustre-MDT0000: Aborting client recovery [ 3333.325922] LustreError: 100252:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3333.326819] Lustre: 100286:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3333.334344] Lustre: 100286:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c706ebfb-b014-459a-b99d-0af5bebd3a85@ [ 3333.339918] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3333.345440] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3333.354783] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3333.402465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505) [ 3333.407166] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1505) [ 3337.866636] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3338.733294] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3359.176468] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 06:33:34 (1788690814) [ 3366.562924] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3368.424326] Lustre: Failing over lustre-MDT0000 [ 3368.742082] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.760676] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 294 previous similar messages [ 3368.810939] Lustre: server umount lustre-MDT0000 complete [ 3380.484136] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3380.486491] LDISKFS-fs (dm-0): recovery complete [ 3380.491694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.876586] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3380.903606] Lustre: lustre-MDT0000: Aborting client recovery [ 3380.909481] LustreError: 102118:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3380.927642] Lustre: 102151:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3380.932430] Lustre: 102151:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 3380.936932] Lustre: 102151:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3380.946874] Lustre: 102151:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3380.953842] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3380.959805] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3380.983901] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3381.031938] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 3381.037700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 3384.717737] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3386.340335] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3399.722067] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3399.727903] Lustre: Skipped 1 previous similar message [ 3408.439247] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 06:34:23 (1788690863) [ 3417.066738] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3419.244672] Lustre: Failing over lustre-MDT0000 [ 3419.570844] Lustre: server umount lustre-MDT0000 complete [ 3433.325137] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3433.327820] LDISKFS-fs (dm-0): recovery complete [ 3433.335177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.715479] Lustre: lustre-MDT0000: Aborting client recovery [ 3433.718526] LustreError: 103985:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3433.723811] Lustre: 104020:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3433.730814] Lustre: 104020:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 3433.735991] Lustre: 104020:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3433.744932] Lustre: 104020:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3433.750319] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3433.756344] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3433.767074] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3433.827946] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 3433.830390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 3438.237495] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3439.088033] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3459.450315] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 06:35:14 (1788690914) [ 3460.624273] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3460.634121] LustreError: 6517:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877844b7b80 x1875574821945984/t201863462916(0) o36->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:43/0 lens 512/456 e 0 to 0 dl 1788690928 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3464.827451] Lustre: Failing over lustre-MDT0000 [ 3465.090784] Lustre: server umount lustre-MDT0000 complete [ 3474.218797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.624560] Lustre: lustre-MDT0000: Aborting client recovery [ 3474.631907] LustreError: 105311:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3474.640592] Lustre: 105344:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3474.647358] Lustre: 105344:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 3474.657284] Lustre: 105344:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3474.666344] Lustre: 105344:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3474.672159] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3474.679203] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3474.691697] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3474.743810] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 3474.759889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1601) [ 3479.139357] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3480.041442] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3499.899598] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3501.522370] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 06:35:56 (1788690956) [ 3509.437125] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3512.259474] Lustre: Failing over lustre-MDT0000 [ 3512.620740] Lustre: server umount lustre-MDT0000 complete [ 3525.670523] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3525.676245] LDISKFS-fs (dm-0): recovery complete [ 3525.686221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.245632] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3526.245961] Lustre: lustre-MDT0000: Aborting client recovery [ 3526.252607] Lustre: Skipped 18 previous similar messages [ 3526.266344] LustreError: 107274:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3526.281168] Lustre: 107306:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3526.287274] Lustre: 107306:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 3526.305797] Lustre: 107306:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3526.321686] Lustre: 107306:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3526.333693] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3526.358504] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3526.383283] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3526.471406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1633) [ 3526.472108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 3531.234918] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3531.315907] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3553.501484] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 06:36:48 (1788691008) [ 3589.616373] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3591.607326] Lustre: Failing over lustre-MDT0000 [ 3591.927662] Lustre: server umount lustre-MDT0000 complete [ 3609.058110] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788691049/real 1788691049] req@ffff9877af1f7800 x1875574843748096/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788691065 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3609.111178] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 3614.539375] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3614.546630] LDISKFS-fs (dm-0): recovery complete [ 3614.558588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.583942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3618.589953] Lustre: Skipped 10 previous similar messages [ 3623.708400] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3624.076650] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3624.077499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3632.916723] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3634.822971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3656.070709] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 06:38:31 (1788691111) [ 3681.678791] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3692.518970] Lustre: Failing over lustre-MDT0000 [ 3693.118886] Lustre: server umount lustre-MDT0000 complete [ 3716.127868] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3716.134651] LDISKFS-fs (dm-0): recovery complete [ 3716.145906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.334852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3727.351586] Lustre: Skipped 41 previous similar messages [ 3727.553792] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3731.232995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3731.242698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3736.920936] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3738.520821] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3760.558809] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 06:40:15 (1788691215) [ 3762.993337] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3763.782746] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.796187] Lustre: Skipped 36 previous similar messages [ 3763.801070] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3763.807980] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3769.546654] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 06:40:24 (1788691224) [ 3794.471506] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3805.113940] Lustre: Failing over lustre-OST0000 [ 3805.214849] Lustre: server umount lustre-OST0000 complete [ 3807.725882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3807.734291] LustreError: Skipped 7 previous similar messages [ 3827.718295] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3827.720777] LDISKFS-fs (dm-2): recovery complete [ 3827.727764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3829.158555] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3829.164067] Lustre: Skipped 3 previous similar messages [ 3830.137803] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3830.150276] Lustre: Skipped 3 previous similar messages [ 3833.658425] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3890.666253] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 06:42:26 (1788691346) [ 3898.481618] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3901.233227] Lustre: Failing over lustre-MDT0000 [ 3901.696235] Lustre: server umount lustre-MDT0000 complete [ 3920.356256] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3920.362903] LustreError: Skipped 7 previous similar messages [ 3924.935170] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3924.937938] LDISKFS-fs (dm-0): recovery complete [ 3924.945690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.592173] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877af3ef480 x1875574844233600/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3930.610479] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 38 previous similar messages [ 3935.919370] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3936.360239] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3936.369528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3945.472773] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3947.015476] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3952.608169] LustreError: 116716:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3952.610336] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3953.632150] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3965.375468] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 06:43:40 (1788691420) [ 3970.658064] LustreError: 6519:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3976.159149] LustreError: 6519:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3976.166677] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnecting [ 3976.205629] LustreError: 8752:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 waking [ 3977.893741] LustreError: 6518:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3983.327568] LustreError: 6518:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3983.343418] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnecting [ 3985.274222] LustreError: 6517:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3990.495163] LustreError: 6517:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3990.509939] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnecting [ 3992.674737] LustreError: 6518:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3998.175176] LustreError: 6518:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4000.220256] LustreError: 6519:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 4005.343164] LustreError: 6519:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4005.355739] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnecting [ 4005.369885] Lustre: Skipped 1 previous similar message [ 4014.674299] LustreError: 56890:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 4014.686224] LustreError: 56890:0:(ldlm_lib.c:1176:target_handle_connect()) Skipped 1 previous similar message [ 4020.192451] LustreError: 56890:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4020.207305] LustreError: 56890:0:(ldlm_lib.c:1176:target_handle_connect()) Skipped 1 previous similar message [ 4027.871374] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnecting [ 4027.883262] Lustre: Skipped 2 previous similar messages [ 4037.835194] LustreError: 8747:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 4037.841872] LustreError: 8747:0:(ldlm_lib.c:1176:target_handle_connect()) Skipped 2 previous similar messages [ 4043.231301] LustreError: 8747:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4043.241242] LustreError: 8747:0:(ldlm_lib.c:1176:target_handle_connect()) Skipped 2 previous similar messages [ 4052.388471] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 06:45:07 (1788691507) [ 4054.343140] LustreError: 6517:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4064.514134] Lustre: lustre-MDT0000: Export ffff9877bd886800 already connecting from 192.168.206.40@tcp [ 4066.567758] Lustre: lustre-MDT0000: Export ffff9877bd886800 already connecting from 192.168.206.40@tcp [ 4069.641992] Lustre: lustre-MDT0000: Export ffff9877bd886800 already connecting from 192.168.206.40@tcp [ 4074.756373] Lustre: lustre-MDT0000: Export ffff9877bd886800 already connecting from 192.168.206.40@tcp [ 4079.876510] Lustre: lustre-MDT0000: Export ffff9877bd886800 already connecting from 192.168.206.40@tcp [ 4079.887639] Lustre: Skipped 1 previous similar message [ 4090.113750] Lustre: lustre-MDT0000: Export ffff9877bd886800 already connecting from 192.168.206.40@tcp [ 4090.131267] Lustre: Skipped 1 previous similar message [ 4094.407150] LustreError: 6517:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4094.417170] Lustre: 6517:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff987688a29f80 x1875574824610688/t0(0) o38->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1788691531 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 4095.232084] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnecting [ 4095.242780] Lustre: Skipped 3 previous similar messages [ 4095.246838] LustreError: 6518:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4119.811431] Lustre: lustre-MDT0000: Export ffff9877bd886800 already connecting from 192.168.206.40@tcp [ 4135.279245] LustreError: 6518:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4135.290191] Lustre: 6518:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff987784375c00 x1875574824615296/t0(0) o38->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1788691571 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4140.296738] LustreError: 8747:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4164.869776] Lustre: lustre-MDT0000: Export ffff9877bd886800 already connecting from 192.168.206.40@tcp [ 4164.878683] Lustre: Skipped 4 previous similar messages [ 4180.343153] LustreError: 8747:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4180.351070] Lustre: 8747:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9877be511f80 x1875574824618880/t0(0) o38->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1788691617 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4185.359834] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnecting [ 4185.379093] Lustre: Skipped 1 previous similar message [ 4185.383823] LustreError: 6517:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4225.407133] LustreError: 6517:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4225.420081] Lustre: 6517:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff98768a377b80 x1875574824622464/t0(0) o38->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1788691662 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4226.307237] LustreError: 8747:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4250.882237] Lustre: lustre-MDT0000: Export ffff9877bd886800 already connecting from 192.168.206.40@tcp [ 4250.890797] Lustre: Skipped 8 previous similar messages [ 4266.351233] LustreError: 8747:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4266.360532] Lustre: 8747:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9877910aa680 x1875574824625664/t0(0) o38->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1788691703 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4271.362665] LustreError: 6519:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4280.667163] LustreError: 6519:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout interrupted [ 4286.024170] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 06:49:01 (1788691741) [ 4294.642260] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4299.286594] Lustre: Failing over lustre-MDT0000 [ 4299.601558] Lustre: server umount lustre-MDT0000 complete [ 4299.756452] LustreError: 7982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.781858] LustreError: 7982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 155 previous similar messages [ 4312.010301] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4312.011906] LDISKFS-fs (dm-0): recovery complete [ 4312.020944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.325230] Lustre: *** cfs_fail_loc=712, val=0*** [ 4312.335914] LustreError: 43910:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9877b48caa00 x1875574844424192/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4312.417888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4312.426544] Lustre: Skipped 3 previous similar messages [ 4312.470682] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4312.471621] Lustre: lustre-MDT0000: Aborting client recovery [ 4312.478632] Lustre: Skipped 6 previous similar messages [ 4312.493120] LustreError: 121569:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4312.498251] Lustre: 121603:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4312.509501] Lustre: 121603:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 4312.514365] Lustre: 121603:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4312.521033] Lustre: 121603:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4312.525712] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4312.531259] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4312.540835] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4312.576647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 4312.590149] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 4317.214900] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4317.672893] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4332.542382] Lustre: Failing over lustre-MDT0000 [ 4332.887126] Lustre: server umount lustre-MDT0000 complete [ 4348.255185] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788691789/real 1788691789] req@ffff9877846bbb80 x1875574844441984/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788691805 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4348.285491] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4353.452909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4364.279907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4364.300983] Lustre: Skipped 15 previous similar messages [ 4364.402037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 4364.412791] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 4365.352731] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4375.625700] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4377.351384] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4386.640470] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 06:50:41 (1788691841) [ 4386.754885] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnecting [ 4386.760885] Lustre: Skipped 2 previous similar messages [ 4394.813816] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 06:50:50 (1788691850) [ 4395.898515] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4395.909644] LustreError: 6527:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff98778444a300 x1875574824689664/t0(0) o700->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:223/0 lens 264/248 e 0 to 0 dl 1788691863 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4415.652745] Lustre: Failing over lustre-MDT0000 [ 4415.904710] Lustre: server umount lustre-MDT0000 complete [ 4420.584050] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4420.585216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4420.606769] Lustre: Skipped 17 previous similar messages [ 4420.630601] LustreError: Skipped 5 previous similar messages [ 4434.807109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4437.762342] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4437.778899] Lustre: Skipped 2 previous similar messages [ 4439.110927] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4440.571547] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4440.590921] Lustre: Skipped 2 previous similar messages [ 4440.664209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 4440.666174] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 4449.864064] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4452.399927] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4468.019697] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 06:52:03 (1788691923) [ 4472.323565] Lustre: Failing over lustre-OST0000 [ 4472.694504] Lustre: server umount lustre-OST0000 complete [ 4492.472179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4499.006616] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4509.603677] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4511.311556] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4584.390864] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 06:53:59 (1788692039) [ 4592.109287] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4594.191197] Lustre: Failing over lustre-MDT0000 [ 4594.482211] Lustre: server umount lustre-MDT0000 complete [ 4610.528648] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4610.549587] LustreError: Skipped 3 previous similar messages [ 4617.154137] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4617.158608] LDISKFS-fs (dm-0): recovery complete [ 4617.173778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4625.983450] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4626.951534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4626.953368] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4698.893777] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 06:55:54 (1788692154) [ 4700.719357] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4700.724674] Lustre: Skipped 2 previous similar messages [ 4712.377514] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 06:56:07 (1788692167) [ 4715.625930] Lustre: Failing over lustre-MDT0000 [ 4715.951592] Lustre: server umount lustre-MDT0000 complete [ 4735.764185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4744.674694] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff987689984e00 x1875574844672896/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4744.703742] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 50 previous similar messages [ 4749.028621] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4750.315978] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4750.319563] LustreError: 129326:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877846bfb80 x1875574824823680/t0(0) o101->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:617/0 lens 328/344 e 0 to 0 dl 1788692257 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4805.378076] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [ 4805.408676] Lustre: 129326:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4805.476750] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 4805.478514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 4811.252787] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4812.915970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4822.314461] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 06:57:57 (1788692277) [ 4824.442204] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4831.783510] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4833.407588] Lustre: Failing over lustre-MDT0000 [ 4833.859643] Lustre: server umount lustre-MDT0000 complete [ 4857.173792] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4857.175792] LDISKFS-fs (dm-0): recovery complete [ 4857.188891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.462535] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8db4bae [ 4869.038461] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4869.206949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3137) [ 4869.206911] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3105) [ 4879.736911] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4881.767885] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4890.149122] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 06:59:05 (1788692345) [ 4891.472959] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4901.540562] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4903.291208] Lustre: Failing over lustre-MDT0000 [ 4903.947466] Lustre: server umount lustre-MDT0000 complete [ 4904.934302] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4904.943033] LustreError: 6517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 175 previous similar messages [ 4926.486843] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4926.489833] LDISKFS-fs (dm-0): recovery complete [ 4926.496439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.914264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4931.920662] Lustre: Skipped 6 previous similar messages [ 4931.969710] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4931.982647] Lustre: Skipped 8 previous similar messages [ 4936.541715] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4937.261056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 4937.264168] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3137) [ 4946.394466] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4948.122530] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4956.623624] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 07:00:11 (1788692411) [ 4957.784725] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4967.221515] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4969.032650] Lustre: Failing over lustre-MDT0000 [ 4969.309206] Lustre: server umount lustre-MDT0000 complete [ 4988.383166] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788692429/real 1788692429] req@ffff987689984a80 x1875574844794368/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788692445 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4988.407580] Lustre: 3646:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4991.060941] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4991.063791] LDISKFS-fs (dm-0): recovery complete [ 4991.075296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.584673] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5004.265302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5004.270636] Lustre: Skipped 28 previous similar messages [ 5004.342083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3201) [ 5004.349129] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 5013.044406] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 07:01:08 (1788692468) [ 5015.123881] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5015.131306] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5015.136279] LustreError: 43600:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877af2dc000 x1875574824887424/t257698037777(0) o35->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:87/0 lens 392/456 e 0 to 0 dl 1788692482 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5017.695790] Lustre: Failing over lustre-MDT0000 [ 5018.051996] Lustre: server umount lustre-MDT0000 complete [ 5036.157741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5046.248643] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8db5efc [ 5047.045059] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5047.061675] Lustre: Skipped 6 previous similar messages [ 5050.990993] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5051.917848] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5051.936828] Lustre: Skipped 6 previous similar messages [ 5051.948877] Lustre: 6520:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877af3ed880 x1875574824887424/t257698037777(0) o35->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:124/0 lens 392/456 e 0 to 0 dl 1788692519 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5051.976496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3233) [ 5051.988964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 5059.658220] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5061.160613] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5068.245396] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 07:02:03 (1788692523) [ 5069.145987] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5069.151607] LustreError: 8747:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff98778444ad80 x1875574824903680/t261993005072(0) o36->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:185/0 lens 504/448 e 0 to 0 dl 1788692580 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5077.030181] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5078.818523] Lustre: Failing over lustre-MDT0000 [ 5079.095592] Lustre: server umount lustre-MDT0000 complete [ 5082.592446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5082.595543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5082.605478] LustreError: Skipped 4 previous similar messages [ 5082.622844] Lustre: Skipped 31 previous similar messages [ 5099.984216] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5099.986954] LDISKFS-fs (dm-0): recovery complete [ 5099.994350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.200640] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5114.917495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 5114.917579] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233) [ 5114.933771] Lustre: 8747:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877b489ce00 x1875574824903680/t261993005072(0) o36->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:231/0 lens 504/2880 e 0 to 0 dl 1788692626 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5122.561579] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5123.922228] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5132.346141] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 07:03:07 (1788692587) [ 5133.590127] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5133.595676] LustreError: 13863:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff98768a55d880 x1875574824920320/t266287972368(0) o36->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:250/0 lens 504/448 e 0 to 0 dl 1788692645 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5135.405302] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5142.346811] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5144.115591] Lustre: Failing over lustre-MDT0000 [ 5144.367348] Lustre: server umount lustre-MDT0000 complete [ 5166.706896] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5166.710976] LDISKFS-fs (dm-0): recovery complete [ 5166.738135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.937727] Lustre: 6518:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877910a8380 x1875574824920320/t266287972368(0) o36->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:294/0 lens 504/2880 e 0 to 0 dl 1788692689 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5177.960496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297) [ 5177.962802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 5177.999111] Lustre: 6518:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5179.536074] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5191.622638] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 07:04:07 (1788692647) [ 5192.711128] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5192.713800] Lustre: Skipped 1 previous similar message [ 5192.721978] LustreError: 6518:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877844b5180 x1875574824934784/t270582939664(0) o36->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:309/0 lens 504/448 e 0 to 0 dl 1788692704 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5192.754996] LustreError: 6518:0:(ldlm_lib.c:3373:target_send_reply_msg()) Skipped 1 previous similar message [ 5194.623080] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5194.628195] Lustre: Skipped 1 previous similar message [ 5202.964924] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5204.781274] Lustre: Failing over lustre-MDT0000 [ 5205.084492] Lustre: server umount lustre-MDT0000 complete [ 5224.927369] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5224.933896] LustreError: Skipped 7 previous similar messages [ 5225.344555] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5225.352723] LDISKFS-fs (dm-0): recovery complete [ 5225.365238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5235.173472] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8db7219 [ 5239.503599] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5240.898348] Lustre: 120060:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877b498a680 x1875574824934784/t270582939664(0) o36->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:357/0 lens 504/2880 e 0 to 0 dl 1788692752 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 5240.906413] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3297) [ 5240.906479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 5248.518557] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 07:05:04 (1788692704) [ 5249.396299] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5251.306461] Lustre: *** cfs_fail_loc=13b, val=315*** [ 5251.314261] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 5251.320321] LustreError: 6521:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877acb83450 x1875574824949504/t274877906960(0) o35->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:324/0 lens 392/456 e 0 to 0 dl 1788692719 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5259.103858] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5260.945536] Lustre: Failing over lustre-MDT0000 [ 5261.288913] Lustre: server umount lustre-MDT0000 complete [ 5282.617905] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5282.619848] LDISKFS-fs (dm-0): recovery complete [ 5282.627852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.593857] Lustre: 6520:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff987785a7fb80 x1875574824949504/t274877906960(0) o35->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:366/0 lens 392/456 e 0 to 0 dl 1788692761 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 5293.605749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 5293.611807] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3329) [ 5293.729529] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5305.169612] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 07:06:00 (1788692760) [ 5306.205137] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 5306.209383] LustreError: 6518:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877b484e300 x1875574824962304/t279172874255(0) o101->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:422/0 lens 664/608 e 0 to 0 dl 1788692817 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5365.510394] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnecting [ 5365.520595] Lustre: Skipped 1 previous similar message [ 5365.532998] Lustre: 6518:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877af2ddc00 x1875574824962304/t279172874255(0) o101->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:482/0 lens 664/3488 e 0 to 0 dl 1788692877 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5372.838857] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 07:07:07 (1788692827) [ 5381.454375] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5383.483855] Lustre: Failing over lustre-MDT0000 [ 5383.834376] Lustre: server umount lustre-MDT0000 complete [ 5406.455136] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5406.457167] LDISKFS-fs (dm-0): recovery complete [ 5406.472248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.316987] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8db7ede [ 5416.970260] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5417.029692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 5417.033695] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3361) [ 5427.162537] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5428.702149] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5446.836759] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 07:08:22 (1788692902) [ 5453.853750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5455.539734] Lustre: Failing over lustre-MDT0000 [ 5455.806374] Lustre: server umount lustre-MDT0000 complete [ 5479.662512] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5479.664371] LDISKFS-fs (dm-0): recovery complete [ 5479.688528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.617388] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5489.705091] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3393) [ 5489.706790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 5500.141338] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5502.465379] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5508.054183] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5520.305325] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 07:09:35 (1788692975) [ 5562.392543] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5564.105769] Lustre: Failing over lustre-MDT0000 [ 5564.516898] Lustre: server umount lustre-MDT0000 complete [ 5566.436171] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.452361] LustreError: 6522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 301 previous similar messages [ 5585.386205] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5585.393451] LDISKFS-fs (dm-0): recovery complete [ 5585.401922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5593.307504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5593.315781] Lustre: Skipped 8 previous similar messages [ 5593.370300] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5593.378232] Lustre: Skipped 8 previous similar messages [ 5597.250629] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5599.019295] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 5599.019900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 5605.867474] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5607.306447] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5657.530758] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 07:11:52 (1788693112) [ 5665.131654] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5667.002798] Lustre: Failing over lustre-MDT0000 [ 5667.284956] Lustre: server umount lustre-MDT0000 complete [ 5686.751259] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788693127/real 1788693127] req@ffff9877b48c3800 x1875574845822720/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788693143 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5686.786633] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 5688.621657] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5688.625385] LDISKFS-fs (dm-0): recovery complete [ 5688.635280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5697.048859] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8df9216 [ 5697.059752] Lustre: Skipped 2 previous similar messages [ 5697.073328] Lustre: MGC192.168.206.140@tcp: Connection restored to 0@lo (at 0@lo) [ 5697.087203] Lustre: Skipped 40 previous similar messages [ 5698.638532] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5698.647130] Lustre: Skipped 7 previous similar messages [ 5701.457776] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5702.706348] Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 5702.713260] Lustre: Skipped 7 previous similar messages [ 5702.745971] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 5702.748485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 5710.822143] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5712.212211] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5721.443719] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 5723.140988] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5729.396583] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 07:13:04 (1788693184) [ 5736.724908] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5799.595538] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5799.604535] Lustre: Skipped 1 previous similar message [ 5799.611224] LustreError: 6519:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877892c7450 x1875574827772160/t296352743435(0) o36->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:161/0 lens 66040/440 e 0 to 0 dl 1788693311 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5860.158933] Lustre: 6517:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff98768829e680 x1875574827772160/t296352743435(0) o36->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:221/0 lens 66040/440 e 0 to 0 dl 1788693371 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5871.151124] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5873.008763] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 07:15:28 (1788693328) [ 5886.087382] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5889.208664] Lustre: Failing over lustre-MDT0000 [ 5889.290438] LustreError: 3646:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9876890c8e00 x1875574845982208/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 5889.606583] Lustre: server umount lustre-MDT0000 complete [ 5892.065048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5892.085104] LustreError: Skipped 6 previous similar messages [ 5892.091642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5892.105678] Lustre: Skipped 32 previous similar messages [ 5907.927160] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5907.945952] LustreError: Skipped 5 previous similar messages [ 5912.660786] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5912.665163] LDISKFS-fs (dm-0): recovery complete [ 5912.686929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5923.242687] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5924.928567] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 5924.930599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 5934.592722] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5936.785544] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5946.457698] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 07:16:41 (1788693401) [ 5967.510517] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5978.851105] Lustre: Failing over lustre-OST0000 [ 5978.927452] Lustre: server umount lustre-OST0000 complete [ 5999.341151] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5999.343900] LDISKFS-fs (dm-2): recovery complete [ 5999.350905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6005.360936] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6019.565315] Lustre: Failing over lustre-OST0000 [ 6019.576144] LustreError: 159042:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [ 6019.584440] Lustre: 158482:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6019.595390] Lustre: 158482:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 6019.608065] Lustre: 158482:0:(ldlm_lib.c:1939:abort_req_replay_queue()) @@@ aborted: req@ffff9877af2dc380 x1875574846059392/t0(17179870645) o6->lustre-MDT0001-mdtlov_UUID@0@lo:331/0 lens 544/0 e 1 to 0 dl 1788693481 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'osp-syn-0-1.0' uid:0 gid:0 projid:4294967295 [ 6019.633865] LustreError: 158482:0:(ofd_obd.c:1325:ofd_iocontrol()) lustre-OST0000: iocontrol from 'tgt_recover_0' cmd=c00866c1 _IOWR('f', 193, 8) unrecognized: rc = -25 [ 6019.644319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6019.867714] Lustre: server umount lustre-OST0000 complete [ 6037.116914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6043.064937] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6046.905711] LustreError: 3643:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 0, old was -19 req@ffff9877b48e5f80 x1875574846059392/t17179870645(17179870645) o6->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 544/432 e 2 to 0 dl 1788693523 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'osp-syn-0-1.0' uid:0 gid:0 projid:4294967295 [ 6052.858917] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6054.716097] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6096.014732] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 07:19:11 (1788693551) [ 6099.025502] Lustre: Failing over lustre-MDT0000 [ 6099.585471] Lustre: server umount lustre-MDT0000 complete [ 6117.981755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6131.620393] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6132.789831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 6132.794421] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 6146.082190] Lustre: Failing over lustre-MDT0000 [ 6146.658949] Lustre: server umount lustre-MDT0000 complete [ 6165.132687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.550629] LustreError: 56890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.569278] LustreError: 56890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 170 previous similar messages [ 6173.666454] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9876890c9f80 x1875574846262656/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6173.689700] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 36 previous similar messages [ 6178.207374] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6179.356173] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 6179.359487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 6188.523585] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6190.404544] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6199.469648] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 07:20:54 (1788693654) [ 6213.687990] Lustre: Failing over lustre-OST0000 [ 6213.846510] Lustre: server umount lustre-OST0000 complete [ 6232.765693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6232.952237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.958771] Lustre: Skipped 6 previous similar messages [ 6232.973611] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6232.983237] Lustre: Skipped 6 previous similar messages [ 6238.850610] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6248.212973] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6249.815851] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6259.740979] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 07:21:54 (1788693714) [ 6261.613927] Lustre: Failing over lustre-MDT0000 [ 6261.878770] Lustre: server umount lustre-MDT0000 complete [ 6272.960803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6273.150359] Lustre: *** cfs_fail_loc=605, val=0*** [ 6273.153635] LustreError: 164886:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0749fc0 failed: rc = -95 [ 6273.164926] LustreError: 164886:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 6273.172306] LustreError: 164886:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 6273.176717] LustreError: 164886:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 6273.182581] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 6273.197222] LustreError: 164886:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 6273.215099] Lustre: server umount lustre-MDT0000 complete [ 6273.219910] LustreError: 164886:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 6282.119960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6289.377354] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8e0d837 [ 6289.384042] LustreError: 165294:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.206.140@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff98768aeebf00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6294.420721] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6295.112243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 6295.122991] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 6302.861818] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 07:22:37 (1788693757) [ 6310.174774] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6313.736225] Lustre: Failing over lustre-MDT0000 [ 6314.151799] Lustre: server umount lustre-MDT0000 complete [ 6331.872228] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788693772/real 1788693772] req@ffff98778444ea00 x1875574846347776/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788693788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6331.902786] Lustre: 3647:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 6336.477725] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6336.479743] LDISKFS-fs (dm-0): recovery complete [ 6336.489876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.515946] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6343.528623] Lustre: Skipped 7 previous similar messages [ 6343.553475] Lustre: *** cfs_fail_loc=707, val=0*** [ 6346.904539] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6347.749897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6347.755922] Lustre: Skipped 25 previous similar messages [ 6403.882127] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 6403.916807] Lustre: 167062:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 6403.925670] Lustre: 167062:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 6404.308946] Lustre: lustre-MDT0000: Recovery over after 1:01, of 2 clients 2 recovered and 0 were evicted. [ 6404.317308] Lustre: Skipped 7 previous similar messages [ 6404.346493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 6404.348184] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 6409.746960] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6411.246231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6419.499118] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 07:24:34 (1788693874) [ 6449.525526] LustreError: 120060:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9877af49d180 x1875574828678784/t0(0) o101->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:12/0 lens 664/0 e 0 to 0 dl 1788693917 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6449.542807] LustreError: 120060:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 6460.601769] LustreError: 120060:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6460.637838] LustreError: 43600:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff987784470e00 x1875574828680064/t0(0) o35->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:23/0 lens 392/0 e 0 to 0 dl 1788693928 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6462.594814] LustreError: 120060:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9877be517100 x1875574828686336/t0(0) o101->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:64/0 lens 576/0 e 0 to 0 dl 1788693969 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 6462.624047] LustreError: 120060:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 6466.018708] LustreError: 43912:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff987784470e00 x1875574846420096/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:28/0 lens 544/0 e 0 to 0 dl 1788693933 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 6466.047185] LustreError: 43912:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 20 previous similar messages [ 6471.140797] LustreError: 8433:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff987784470700 x1875574846422784/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:33/0 lens 544/0 e 0 to 0 dl 1788693938 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 6471.174838] LustreError: 8433:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 7 previous similar messages [ 6477.358510] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 07:25:32 (1788693932) [ 6506.409663] LustreError: 8438:0:(tgt_handler.c:2827:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 6517.439141] LustreError: 8438:0:(tgt_handler.c:2827:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 6524.588789] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 07:26:19 (1788693979) [ 6552.086933] LustreError: 13863:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9877be511f80 x1875574828705792/t0(0) o101->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:114/0 lens 576/0 e 0 to 0 dl 1788694019 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6552.111732] LustreError: 13863:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 6557.215358] LustreError: 13863:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6568.425442] LustreError: 43922:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9876876c6300 x1875574846468096/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:131/0 lens 544/0 e 0 to 0 dl 1788694036 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 6568.456957] LustreError: 43922:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 6569.711134] LustreError: 6517:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6587.047496] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 07:27:22 (1788694042) [ 6679.032967] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 07:28:54 (1788694134) [ 6707.097895] LustreError: 8747:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff98768687e680 x1875574828788352/t0(0) o101->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:269/0 lens 576/0 e 0 to 0 dl 1788694174 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6707.124037] LustreError: 8747:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 99 previous similar messages [ 6707.135553] LustreError: 8747:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6707.140807] LustreError: 8747:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 6707.559119] LustreError: 8747:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6723.393764] LustreError: 43600:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6723.403420] LustreError: 43600:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 6723.831344] LustreError: 43600:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6723.842537] LustreError: 43600:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 6756.699818] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 07:30:12 (1788694212) [ 6791.499498] Lustre: DEBUG MARKER: phase 2 [ 6800.158638] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 07:30:55 (1788694255) [ 6883.190292] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 07:32:18 (1788694338) [ 6884.861712] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6886.989691] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 07:32:22 (1788694342) [ 6892.154562] Lustre: DEBUG MARKER: Started rundbench load pid=135394 ... [ 6900.888803] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6903.394731] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6905.247529] Lustre: Failing over lustre-MDT0000 [ 6905.449936] Lustre: server umount lustre-MDT0000 complete [ 6905.825327] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6905.830383] LustreError: 8747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6905.847758] Lustre: Skipped 23 previous similar messages [ 6905.886306] LustreError: 8747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 88 previous similar messages [ 6926.968139] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 6926.971238] LDISKFS-fs (dm-0): recovery complete [ 6926.978341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6927.085299] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6927.092328] LustreError: Skipped 4 previous similar messages [ 6927.371488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6927.376271] Lustre: Skipped 2 previous similar messages [ 6927.456693] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6927.471109] Lustre: Skipped 2 previous similar messages [ 6931.576242] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6932.553855] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) [ 6932.559870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) [ 6941.453984] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6943.480183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6954.881851] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6957.510403] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6959.565197] Lustre: Failing over lustre-MDT0001 [ 6959.604424] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [ 6959.757437] Lustre: server umount lustre-MDT0001 complete [ 6980.666042] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 6980.671267] LDISKFS-fs (dm-1): recovery complete [ 6980.687535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6982.161371] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6982.171025] Lustre: Skipped 1 previous similar message [ 6983.795758] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6986.216151] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6986.219909] Lustre: Skipped 7 previous similar messages [ 6986.930522] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:549 to 0x280000400:577) [ 6986.931580] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:533 to 0x2c0000400:577) [ 6991.371830] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6992.753501] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7002.078103] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7004.435943] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 7005.934941] Lustre: Failing over lustre-MDT0000 [ 7006.156928] Lustre: server umount lustre-MDT0000 complete [ 7006.687782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7006.693128] LustreError: Skipped 3 previous similar messages [ 7023.072098] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788694463/real 1788694463] req@ffff9877af49d500 x1875574847030912/t0(0) o400->MGC192.168.206.140@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788694479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7025.837243] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7025.839670] LDISKFS-fs (dm-0): recovery complete [ 7025.845471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7032.296292] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a8e33834 [ 7036.605611] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7037.968795] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7037.975579] Lustre: Skipped 2 previous similar messages [ 7038.003056] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) [ 7038.003816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) [ 7043.953277] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7045.124251] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7051.426917] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 07:35:07 (1788694507) [ 7177.861219] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7190.029542] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 7191.766758] Lustre: Failing over lustre-MDT0000 [ 7191.787127] LustreError: 3647:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff987689d4c000 x1875574848578688/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 7191.808536] LustreError: 3647:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 7191.838195] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 7194.077934] Lustre: server umount lustre-MDT0000 complete [ 7213.759653] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7213.762167] LDISKFS-fs (dm-0): recovery complete [ 7213.768828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7222.752259] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98768687d880 x1875574848591232/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7225.981793] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7231.616442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5437 to 0x280000401:5473) [ 7231.616465] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5373 to 0x2c0000401:5409) [ 7236.485728] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7237.896833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7338.046407] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 07:39:53 (1788694793) [ 7464.933695] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7476.547990] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 7478.114856] Lustre: Failing over lustre-MDT0000 [ 7478.172560] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 7484.079763] Lustre: server umount lustre-MDT0000 complete [ 7503.277972] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7503.280085] LDISKFS-fs (dm-0): recovery complete [ 7503.285295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7505.893428] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7505.911702] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 133 previous similar messages [ 7514.353985] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7518.499217] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5688 to 0x2c0000401:5729) [ 7518.499484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5753 to 0x280000401:5793) [ 7523.287578] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7524.708880] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7530.920543] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 07:43:06 (1788694986) [ 7656.624707] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7667.957341] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 7669.160630] Lustre: Failing over lustre-MDT0001 [ 7669.177131] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 7669.183272] LustreError: Skipped 1 previous similar message [ 7669.185562] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.192412] Lustre: Skipped 15 previous similar messages [ 7669.195156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7669.198715] Lustre: Skipped 5 previous similar messages [ 7669.336241] Lustre: server umount lustre-MDT0001 complete [ 7686.543415] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7686.546291] LDISKFS-fs (dm-1): recovery complete [ 7686.557585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.792473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7686.798581] Lustre: Skipped 4 previous similar messages [ 7686.814261] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7686.820522] Lustre: Skipped 4 previous similar messages [ 7688.396577] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7688.400602] Lustre: Skipped 3 previous similar messages [ 7689.192401] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7692.268724] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7692.274708] Lustre: Skipped 15 previous similar messages [ 7692.288399] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7692.294831] Lustre: Skipped 2 previous similar messages [ 7692.326969] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1237 to 0x280000400:1313) [ 7692.327191] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1236 to 0x2c0000400:1281) [ 7695.971496] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7697.110432] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7824.153501] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7835.542703] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 7836.799520] Lustre: Failing over lustre-MDT0001 [ 7836.856945] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7837.061666] Lustre: server umount lustre-MDT0001 complete [ 7853.756571] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7853.758949] LDISKFS-fs (dm-1): recovery complete [ 7853.765408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7856.158570] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7859.206613] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1237 to 0x280000400:1345) [ 7859.206726] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1236 to 0x2c0000400:1313) [ 7862.199890] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7863.137585] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7868.622296] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 07:48:44 (1788695324) [ 7876.790524] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7878.837285] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7880.048659] Lustre: Failing over lustre-OST0000 [ 7880.104554] Lustre: server umount lustre-OST0000 complete [ 7896.254615] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7896.256259] LDISKFS-fs (dm-2): recovery complete [ 7896.268595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7899.722702] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7905.015451] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7905.898058] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7916.360534] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7918.115255] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 7918.948363] Lustre: Failing over lustre-OST0000 [ 7918.994768] Lustre: server umount lustre-OST0000 complete [ 7934.919875] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7934.921267] LDISKFS-fs (dm-2): recovery complete [ 7934.925482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7935.012965] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7938.026125] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7944.012124] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7945.145344] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7952.667854] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 07:50:08 (1788695408) [ 8076.812092] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8080.374211] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8091.515483] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 8092.548950] Lustre: Failing over lustre-MDT0000 [ 8092.562202] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 8092.777673] Lustre: server umount lustre-MDT0000 complete [ 8094.703034] LustreError: 7458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788695551 with bad export cookie 8072999297524619158 [ 8094.705591] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8094.705830] Lustre: Failing over lustre-MDT0001 [ 8094.710699] LustreError: 7458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8094.722128] LustreError: Skipped 3 previous similar messages [ 8094.906158] Lustre: server umount lustre-MDT0001 complete [ 8112.309743] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8112.313120] LDISKFS-fs (dm-0): recovery complete [ 8112.321978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8112.460693] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 8112.462972] LDISKFS-fs (dm-1): recovery complete [ 8112.470782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8115.168552] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788695555/real 1788695555] req@ffff98768a3fed80 x1875574858095104/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788695571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8115.186958] Lustre: 3644:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 8119.327580] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877bf4c5f80 x1875574858097536/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8119.525729] LustreError: 179137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8119.535742] LustreError: 179137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 56 previous similar messages [ 8122.002135] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8122.076459] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8123.109811] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1237 to 0x280000400:1377) [ 8123.109828] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1236 to 0x2c0000400:1345) [ 8137.723898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7117 to 0x280000401:7137) [ 8137.724377] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7053 to 0x2c0000401:7073) [ 8141.101541] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8141.914462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8142.733561] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8147.597486] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 07:53:23 (1788695603) [ 8151.574356] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8152.916091] Lustre: Failing over lustre-MDT0000 [ 8153.073749] Lustre: server umount lustre-MDT0000 complete [ 8169.649906] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8169.651718] LDISKFS-fs (dm-0): recovery complete [ 8169.659503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8172.078867] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8173.314949] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 8189.716463] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 8189.819105] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7053 to 0x2c0000401:7105) [ 8189.821241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7169) [ 8193.066548] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8193.882701] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8198.640566] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 07:54:14 (1788695654) [ 8202.696590] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8204.258318] Lustre: Failing over lustre-MDT0000 [ 8204.437599] Lustre: server umount lustre-MDT0000 complete [ 8220.754803] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8220.757783] LDISKFS-fs (dm-0): recovery complete [ 8220.766697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8222.866398] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8224.395543] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 8224.398498] LustreError: 193350:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff987689286a00 x1875574847316608/t335007462925(335007462925) o101->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:277/0 lens 520/664 e 0 to 0 dl 1788695692 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 8240.918545] Lustre: lustre-MDT0000: Client c706ebfb-b014-459a-b99d-0af5bebd3a85 (at 192.168.206.40@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 8240.929557] Lustre: 193350:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877be510000 x1875574847316608/t335007462925(335007462925) o101->c706ebfb-b014-459a-b99d-0af5bebd3a85@192.168.206.40@tcp:293/0 lens 520/3488 e 0 to 0 dl 1788695708 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 8240.988590] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7107 to 0x2c0000401:7137) [ 8240.992690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7201) [ 8244.258882] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8245.059868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8249.764330] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 07:55:05 (1788695705) [ 8251.435813] Lustre: Failing over lustre-OST0000 [ 8251.501854] Lustre: server umount lustre-OST0000 complete [ 8253.211683] Lustre: Failing over lustre-MDT0000 [ 8253.300421] Lustre: server umount lustre-MDT0000 complete [ 8267.465355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.664922] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8270.715305] Lustre: lustre-MDT0000: Denying connection for new client ed0c3242-f689-4f41-80e5-860fbd35ae6a (at 192.168.206.40@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 8272.893652] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7107 to 0x2c0000401:7169) [ 8279.196209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8282.479696] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8284.651818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7233) [ 8288.238290] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 07:55:44 (1788695744) [ 8288.782864] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8288.786081] LustreError: 193353:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877b5698380 x1875574858913792/t352187318279(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:341/0 lens 1312/4320 e 0 to 0 dl 1788695756 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8292.474575] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8293.460632] Lustre: Failing over lustre-MDT0000 [ 8293.694824] Lustre: server umount lustre-MDT0000 complete [ 8295.906061] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8295.913667] Lustre: Skipped 30 previous similar messages [ 8309.938383] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8309.941165] LDISKFS-fs (dm-0): recovery complete [ 8309.949639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8310.151165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8310.154361] Lustre: Skipped 8 previous similar messages [ 8310.171397] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8310.176016] Lustre: Skipped 9 previous similar messages [ 8311.552362] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8311.555886] Lustre: Skipped 9 previous similar messages [ 8312.236257] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8315.362403] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 8315.368854] Lustre: Skipped 28 previous similar messages [ 8315.381241] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8315.393095] Lustre: Skipped 9 previous similar messages [ 8315.418510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7265) [ 8315.418572] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7201) [ 8318.611423] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8319.581697] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8325.107877] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 07:56:20 (1788695780) [ 8325.742487] LustreError: 194356:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877af7c6300 x1875574858945792/t356482285580(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:378/0 lens 2520/4320 e 0 to 0 dl 1788695793 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8329.771504] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8334.252076] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8335.305846] Lustre: Failing over lustre-MDT0001 [ 8335.310596] LustreError: 203603:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x2000284a1:0x4:0x0].0x0 (ffff987787b9b700) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8335.322589] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [ 8335.840790] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8335.848923] LustreError: Skipped 7 previous similar messages [ 8341.126870] Lustre: server umount lustre-MDT0001 complete [ 8357.062138] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8357.068239] LDISKFS-fs (dm-1): recovery complete [ 8357.077160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8358.936570] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8362.501693] Lustre: 194210:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877af197800 x1875574850379776/t21474837775(0) o36->ed0c3242-f689-4f41-80e5-860fbd35ae6a@192.168.206.40@tcp:442/0 lens 560/2880 e 0 to 0 dl 1788695857 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8362.501933] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1356 to 0x2c0000400:1377) [ 8362.503454] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1388 to 0x280000400:1409) [ 8365.214810] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8365.974759] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8370.774586] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 07:57:06 (1788695826) [ 8371.301149] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8371.303142] Lustre: Skipped 1 previous similar message [ 8374.864199] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8378.279700] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8379.162973] Lustre: Failing over lustre-MDT0000 [ 8379.431615] Lustre: server umount lustre-MDT0000 complete [ 8395.349890] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8395.355506] LDISKFS-fs (dm-0): recovery complete [ 8395.363649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8397.699086] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8400.978173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7297) [ 8400.978254] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7233) [ 8403.622435] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8404.292943] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8406.611472] Lustre: Failing over lustre-MDT0001 [ 8406.743190] Lustre: server umount lustre-MDT0001 complete [ 8423.246537] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8423.248784] LDISKFS-fs (dm-1): recovery complete [ 8423.256318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8425.376737] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8428.537269] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1388 to 0x2c0000400:1409) [ 8428.538817] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1420 to 0x280000400:1441) [ 8430.801270] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8431.473163] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8436.559412] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 07:58:12 (1788695892) [ 8437.095965] LustreError: 194356:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877bf1b6300 x1875574859036160/t360777252880(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:489/0 lens 2520/4320 e 0 to 0 dl 1788695904 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8437.113112] LustreError: 194356:0:(ldlm_lib.c:3373:target_send_reply_msg()) Skipped 1 previous similar message [ 8443.982699] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8448.039933] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8449.044426] Lustre: Failing over lustre-MDT0000 [ 8449.206434] Lustre: server umount lustre-MDT0000 complete [ 8451.046978] LustreError: 151888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788695907 with bad export cookie 8072999297525757456 [ 8451.049367] Lustre: Failing over lustre-MDT0001 [ 8451.052579] LustreError: 151888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8451.055355] LustreError: 209863:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x2000284a1:0xa:0x0].0x0 (ffff98778a23f900) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8451.077352] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [ 8451.081400] Lustre: Skipped 7 previous similar messages [ 8457.184119] LustreError: 209863:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9877bf1b5880 x1875574859049856/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8457.194155] LustreError: 209863:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 8457.198638] LustreError: 209863:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8457.314256] Lustre: server umount lustre-MDT0001 complete [ 8474.047470] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8474.049035] LDISKFS-fs (dm-0): recovery complete [ 8474.058058] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8474.061428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8474.065465] LDISKFS-fs (dm-1): recovery complete [ 8474.074626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8474.225957] LustreError: 210965:0:(llog.c:1655:llog_backup()) MGC192.168.206.140@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8474.230349] Lustre: 210965:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.140@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8477.996086] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8478.179421] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8481.112501] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7265) [ 8481.112621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7329) [ 8481.144140] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1420 to 0x2c0000400:1441) [ 8481.145621] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1452 to 0x280000400:1473) [ 8481.151914] Lustre: 210974:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877bea68380 x1875574850450688/t30064771142(0) o36->ed0c3242-f689-4f41-80e5-860fbd35ae6a@192.168.206.40@tcp:559/0 lens 560/2880 e 0 to 0 dl 1788695974 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8483.922933] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8484.591493] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8485.293158] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8489.795436] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 07:59:05 (1788695945) [ 8496.391698] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8497.161665] Lustre: Failing over lustre-MDT0000 [ 8497.460712] Lustre: server umount lustre-MDT0000 complete [ 8505.615306] Lustre: lustre-MDT0001: Client ed0c3242-f689-4f41-80e5-860fbd35ae6a (at 192.168.206.40@tcp) reconnecting [ 8505.619285] Lustre: Skipped 2 previous similar messages [ 8505.625528] Lustre: 210973:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877af197b80 x1875574850484864/t34359738438(0) o36->ed0c3242-f689-4f41-80e5-860fbd35ae6a@192.168.206.40@tcp:558/0 lens 560/2880 e 0 to 0 dl 1788695973 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8512.468912] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8512.470658] LDISKFS-fs (dm-0): recovery complete [ 8512.475190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8514.302583] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8518.150470] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7297) [ 8518.150473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7361) [ 8520.574357] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8521.306475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8525.493833] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 07:59:41 (1788695981) [ 8525.947258] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8525.950553] Lustre: Skipped 2 previous similar messages [ 8529.328427] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8530.094731] Lustre: Failing over lustre-MDT0001 [ 8530.194374] Lustre: server umount lustre-MDT0001 complete [ 8545.339556] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8545.341780] LDISKFS-fs (dm-1): recovery complete [ 8545.351642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8547.337676] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8550.908848] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1494 to 0x280000400:1537) [ 8550.908853] Lustre: 212046:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9877b48e5180 x1875574850516608/t34359738509(0) o36->ed0c3242-f689-4f41-80e5-860fbd35ae6a@192.168.206.40@tcp:603/0 lens 560/2880 e 0 to 0 dl 1788696018 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8550.911116] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1462 to 0x2c0000400:1505) [ 8553.360503] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8554.037740] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8558.598974] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 08:00:14 (1788696014) [ 8565.474640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8569.339913] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8570.315140] Lustre: Failing over lustre-MDT0000 [ 8570.521620] Lustre: server umount lustre-MDT0000 complete [ 8586.120804] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8586.123124] LDISKFS-fs (dm-0): recovery complete [ 8586.128794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.965057] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8591.368758] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7329) [ 8591.368922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7393) [ 8594.041991] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8594.724584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8596.940419] Lustre: Failing over lustre-MDT0001 [ 8597.058937] Lustre: server umount lustre-MDT0001 complete [ 8612.653812] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8612.656053] LDISKFS-fs (dm-1): recovery complete [ 8612.664589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8614.620176] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8617.981386] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1516 to 0x2c0000400:1537) [ 8617.982196] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1548 to 0x280000400:1569) [ 8620.790321] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8621.497346] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8626.597773] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 08:01:22 (1788696082) [ 8627.168534] LustreError: 213404:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff987691633100 x1875574850583936/t42949673030(0) o36->ed0c3242-f689-4f41-80e5-860fbd35ae6a@192.168.206.40@tcp:679/0 lens 560/448 e 0 to 0 dl 1788696094 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8627.183835] LustreError: 213404:0:(ldlm_lib.c:3373:target_send_reply_msg()) Skipped 3 previous similar messages [ 8633.821075] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8637.442769] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8638.442228] Lustre: Failing over lustre-MDT0000 [ 8638.699434] Lustre: server umount lustre-MDT0000 complete [ 8640.472905] LustreError: 151887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696097 with bad export cookie 8072999297525771232 [ 8640.475444] Lustre: Failing over lustre-MDT0001 [ 8640.481154] LustreError: 151887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8640.625957] Lustre: server umount lustre-MDT0001 complete [ 8657.342118] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8657.344909] LDISKFS-fs (dm-0): recovery complete [ 8657.345758] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8657.352100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8657.358123] LDISKFS-fs (dm-1): recovery complete [ 8657.371868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.887425] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98768b0fb480 x1875574859216512/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8668.473717] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8668.596264] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8669.437087] Lustre: 222129:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff987784201f80 x1875574850583936/t42949673030(0) o36->ed0c3242-f689-4f41-80e5-860fbd35ae6a@192.168.206.40@tcp:722/0 lens 560/2880 e 0 to 0 dl 1788696137 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8669.439233] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1580 to 0x280000400:1601) [ 8669.454451] Lustre: 222129:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8669.458258] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1548 to 0x2c0000400:1569) [ 8679.224385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7425) [ 8679.224514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7361) [ 8682.416670] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8683.242212] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8684.048501] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8688.871961] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 08:02:24 (1788696144) [ 8692.834308] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8693.723785] Lustre: Failing over lustre-MDT0001 [ 8693.728980] LustreError: 223935:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x2000284a1:0x17:0x0].0xf7117594 (ffff9877914c0300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8693.736520] LustreError: 223935:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 1 previous similar message [ 8693.754793] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [ 8693.760899] Lustre: Skipped 5 previous similar messages [ 8700.012168] Lustre: server umount lustre-MDT0001 complete [ 8715.750318] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8715.752132] LDISKFS-fs (dm-1): recovery complete [ 8715.762782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.612783] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8721.410719] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1580 to 0x2c0000400:1601) [ 8721.410883] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1612 to 0x280000400:1633) [ 8723.770133] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8724.441308] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8728.579427] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 08:03:04 (1788696184) [ 8732.634307] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8733.517237] Lustre: Failing over lustre-MDT0000 [ 8733.808779] Lustre: server umount lustre-MDT0000 complete [ 8736.736077] LustreError: 179137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8736.744419] LustreError: 179137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 188 previous similar messages [ 8744.415155] Lustre: 224563:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788696185/real 1788696185] req@ffff987784206d80 x1875574859281664/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1488/4320 e 0 to 1 dl 1788696201 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8744.429684] Lustre: 224563:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 31 previous similar messages [ 8749.021855] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8749.023633] LDISKFS-fs (dm-0): recovery complete [ 8749.029174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8749.113750] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8749.118913] LustreError: Skipped 9 previous similar messages [ 8750.850966] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8754.694355] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7393) [ 8754.694686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7457) [ 8756.940732] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8757.585151] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8762.012679] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 08:03:38 (1788696218) [ 8765.998370] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8769.716913] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8770.714877] Lustre: Failing over lustre-MDT0000 [ 8770.992640] Lustre: server umount lustre-MDT0000 complete [ 8786.854335] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8786.858748] LDISKFS-fs (dm-0): recovery complete [ 8786.868177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8789.080675] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8792.592368] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7425) [ 8792.593053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7489) [ 8795.317460] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8796.127564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8798.537701] Lustre: Failing over lustre-MDT0001 [ 8798.662310] Lustre: server umount lustre-MDT0001 complete [ 8814.590597] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8814.596418] LDISKFS-fs (dm-1): recovery complete [ 8814.605995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8816.751240] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8820.226824] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1580 to 0x2c0000400:1633) [ 8820.227347] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1612 to 0x280000400:1665) [ 8823.129989] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8823.898799] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8828.940726] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 08:04:44 (1788696284) [ 8829.522658] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8829.525242] Lustre: Skipped 5 previous similar messages [ 8833.161983] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8837.195595] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8838.195178] Lustre: Failing over lustre-MDT0000 [ 8838.490353] Lustre: server umount lustre-MDT0000 complete [ 8840.322762] LustreError: 151888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696297 with bad export cookie 8072999297525781676 [ 8840.326686] Lustre: Failing over lustre-MDT0001 [ 8840.332590] LustreError: 151888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8840.347735] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [ 8840.350713] Lustre: Skipped 5 previous similar messages [ 8845.791682] LustreError: 232172:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9877b76d6300 x1875574859368064/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8845.808781] LustreError: 232172:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8845.942157] Lustre: server umount lustre-MDT0001 complete [ 8861.507659] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8861.509466] LDISKFS-fs (dm-1): recovery complete [ 8861.520068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.598109] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8861.600548] LDISKFS-fs (dm-0): recovery complete [ 8861.603905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8861.624067] LustreError: 233251:0:(llog.c:1655:llog_backup()) MGC192.168.206.140@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8861.630686] Lustre: 233251:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.140@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8865.250889] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x70090e16a9009e6f [ 8867.271935] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8867.400272] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8871.567079] Lustre: 233280:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff98768b0fa680 x1875574850664192/t55834574850(0) o36->ed0c3242-f689-4f41-80e5-860fbd35ae6a@192.168.206.40@tcp:194/0 lens 496/2888 e 0 to 0 dl 1788696364 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8871.568428] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1580 to 0x2c0000400:1665) [ 8871.568428] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1612 to 0x280000400:1697) [ 8871.580802] Lustre: 233280:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8881.984575] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7457) [ 8881.984659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7521) [ 8885.246153] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8886.178868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8887.073773] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8892.203801] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 08:05:48 (1788696348) [ 8892.820978] LustreError: 233280:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff9877af49f100 x1875574850679936/t60129542146(0) o36->ed0c3242-f689-4f41-80e5-860fbd35ae6a@192.168.206.40@tcp:190/0 lens 496/456 e 0 to 0 dl 1788696360 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8892.842067] LustreError: 233280:0:(ldlm_lib.c:3373:target_send_reply_msg()) Skipped 4 previous similar messages [ 8897.167957] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8898.083438] Lustre: Failing over lustre-MDT0000 [ 8898.229437] Lustre: server umount lustre-MDT0000 complete [ 8902.631624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8902.644438] Lustre: Skipped 52 previous similar messages [ 8909.060642] Lustre: lustre-MDT0001: Client ed0c3242-f689-4f41-80e5-860fbd35ae6a (at 192.168.206.40@tcp) reconnecting [ 8909.064107] Lustre: Skipped 1 previous similar message [ 8913.934182] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8913.936326] LDISKFS-fs (dm-0): recovery complete [ 8913.940544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8914.113355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8914.117894] Lustre: Skipped 17 previous similar messages [ 8914.140523] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8914.147119] Lustre: Skipped 17 previous similar messages [ 8914.179301] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8914.186225] Lustre: Skipped 17 previous similar messages [ 8915.839831] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8919.522359] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 8919.528775] Lustre: Skipped 60 previous similar messages [ 8919.540930] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8919.545991] Lustre: Skipped 17 previous similar messages [ 8919.561417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7553) [ 8919.561418] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7489) [ 8921.800680] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8922.456603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8926.789688] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 08:06:22 (1788696382) [ 8930.440510] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8931.232453] Lustre: Failing over lustre-MDT0001 [ 8931.331257] Lustre: server umount lustre-MDT0001 complete [ 8946.622927] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8946.624877] LDISKFS-fs (dm-1): recovery complete [ 8946.635173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8948.572440] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8952.309503] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1612 to 0x280000400:1729) [ 8952.309512] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1580 to 0x2c0000400:1697) [ 8954.541314] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8955.208184] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8959.849465] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 08:06:55 (1788696415) [ 8964.053109] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8967.686828] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8968.584185] Lustre: Failing over lustre-MDT0000 [ 8968.728184] Lustre: server umount lustre-MDT0000 complete [ 8972.259549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8972.265964] LustreError: Skipped 15 previous similar messages [ 8983.854101] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8983.856177] LDISKFS-fs (dm-0): recovery complete [ 8983.860355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8985.703457] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8989.188051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7585) [ 8989.188061] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7521) [ 8991.411372] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8992.129633] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8994.366156] Lustre: Failing over lustre-MDT0001 [ 8994.476969] Lustre: server umount lustre-MDT0001 complete [ 9010.398109] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9010.400062] LDISKFS-fs (dm-1): recovery complete [ 9010.407091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.512925] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9015.812371] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1612 to 0x280000400:1761) [ 9015.814128] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1580 to 0x2c0000400:1729) [ 9018.417241] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9019.107286] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9023.327365] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 08:07:59 (1788696479) [ 9027.201628] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9030.668424] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9031.552854] Lustre: Failing over lustre-MDT0000 [ 9031.718467] Lustre: server umount lustre-MDT0000 complete [ 9033.533479] LustreError: 151888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696490 with bad export cookie 8072999297525789257 [ 9033.535810] Lustre: Failing over lustre-MDT0001 [ 9033.539101] LustreError: 151888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9033.692057] Lustre: server umount lustre-MDT0001 complete [ 9049.622991] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9049.627151] LDISKFS-fs (dm-1): recovery complete [ 9049.634583] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9049.636444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9049.639228] LDISKFS-fs (dm-0): recovery complete [ 9049.647898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9049.749608] LustreError: 244471:0:(llog.c:1655:llog_backup()) MGC192.168.206.140@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 9049.754174] Lustre: 244471:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.140@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 9060.622333] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9060.756326] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9065.085362] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1612 to 0x280000400:1793) [ 9065.085725] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1580 to 0x2c0000400:1761) [ 9065.131114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7139 to 0x280000401:7617) [ 9065.131114] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7171 to 0x2c0000401:7553) [ 9067.670464] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9068.425345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9069.161547] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9073.527924] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 08:08:49 (1788696529) [ 9074.284226] Lustre: 245835:0:(genops.c:1774:obd_export_evict_by_uuid()) lustre-MDT0000: evicting ed0c3242-f689-4f41-80e5-860fbd35ae6a at adminstrative request [ 9078.445995] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 08:08:54 (1788696534) [ 9081.933281] Lustre: Failing over lustre-MDT0000 [ 9082.110353] Lustre: server umount lustre-MDT0000 complete [ 9096.208447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.295194] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9101.886699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7668 to 0x280000401:7713) [ 9101.893102] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7605 to 0x2c0000401:7649) [ 9104.456951] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9105.178738] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9109.587583] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 08:09:25 (1788696565) [ 9116.908310] Lustre: Failing over lustre-OST0000 [ 9116.974491] Lustre: server umount lustre-OST0000 complete [ 9130.525240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9132.934566] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9137.218923] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9137.863188] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9141.788517] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 08:09:57 (1788696597) [ 9143.655479] Lustre: Failing over lustre-MDT0000 [ 9143.852491] Lustre: server umount lustre-MDT0000 complete [ 9144.803804] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696601 with bad export cookie 8072999297525801990 [ 9146.846465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9148.817540] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9149.731726] Lustre: lustre-MDT0000: Denying connection for new client f7e65ff0-4975-4d74-8ad4-775732dc5c8e (at 192.168.206.40@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 9149.741149] Lustre: Skipped 1 previous similar message [ 9152.507020] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7605 to 0x2c0000401:7681) [ 9152.507193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7668 to 0x280000401:7745) [ 9157.735637] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 08:10:13 (1788696613) [ 9161.306360] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9162.515671] Lustre: Failing over lustre-OST0000 [ 9162.558958] Lustre: server umount lustre-OST0000 complete [ 9177.547048] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9177.548850] LDISKFS-fs (dm-2): recovery complete [ 9177.554341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9180.025496] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9184.174026] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9184.843937] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9188.878362] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 08:10:45 (1788696645) [ 9192.156955] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9194.316867] Lustre: Failing over lustre-OST0000 [ 9194.375369] Lustre: server umount lustre-OST0000 complete [ 9209.312882] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9209.315093] LDISKFS-fs (dm-2): recovery complete [ 9209.319471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9211.011979] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.206.40@tcp inode [0x20002f201:0x5:0x0] object 0x280000401:7747 extent [0-4194303]: client csum 35a2fd80, server csum 8501fd6a [ 9211.668259] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9215.868035] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9216.552703] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9220.808194] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 08:11:16 (1788696676) [ 9223.799075] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9226.491977] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9229.135249] Lustre: Failing over lustre-MDT0000 [ 9229.279863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9229.282705] Lustre: Skipped 5 previous similar messages [ 9229.419996] Lustre: server umount lustre-MDT0000 complete [ 9231.143392] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788696687 with bad export cookie 8072999297525816214 [ 9231.145465] Lustre: Failing over lustre-OST0000 [ 9231.149061] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9231.206771] Lustre: server umount lustre-OST0000 complete [ 9247.185164] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9247.187100] LDISKFS-fs (dm-0): recovery complete [ 9247.193114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.391485] LustreError: 3643:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9877bf090e00 x1875574859669248/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9257.163463] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9260.746430] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7605 to 0x2c0000401:7713) [ 9272.761037] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9272.762963] LDISKFS-fs (dm-2): recovery complete [ 9272.768301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9275.228409] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9282.967485] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 08:12:18 (1788696738) [ 9288.777423] Lustre: Failing over lustre-OST0000 [ 9288.840356] Lustre: server umount lustre-OST0000 complete [ 9290.428167] Lustre: Failing over lustre-MDT0000 [ 9290.706463] Lustre: server umount lustre-MDT0000 complete [ 9304.129595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9305.890821] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9309.693367] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7605 to 0x2c0000401:7745) [ 9312.178650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9314.450246] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9315.371398] Lustre: lustre-OST0000: Denying connection for new client 90b009ae-75cc-480a-a0b5-dbdb11b187af (at 192.168.206.40@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 9320.703299] Lustre: lustre-OST0000: Denying connection for new client 90b009ae-75cc-480a-a0b5-dbdb11b187af (at 192.168.206.40@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 9330.942706] Lustre: lustre-OST0000: Denying connection for new client 90b009ae-75cc-480a-a0b5-dbdb11b187af (at 192.168.206.40@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 9330.949770] Lustre: Skipped 1 previous similar message [ 9351.422178] Lustre: lustre-OST0000: Denying connection for new client 90b009ae-75cc-480a-a0b5-dbdb11b187af (at 192.168.206.40@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:33 [ 9351.428128] Lustre: Skipped 3 previous similar messages [ 9384.500180] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9384.503304] Lustre: 259354:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client f7e65ff0-4975-4d74-8ad4-775732dc5c8e@ [ 9384.508137] Lustre: 259354:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 9384.513981] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9384.533708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7796 to 0x280000401:7817) [ 9388.123583] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 9396.258064] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 9398.424939] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 08:14:14 (1788696854) [ 9399.892064] Lustre: Failing over lustre-OST0000 [ 9399.931412] Lustre: server umount lustre-OST0000 complete [ 9402.624411] LustreError: 179162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9402.630773] LustreError: 179162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 244 previous similar messages [ 9413.524345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9415.734692] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9419.892464] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 08:14:36 (1788696876) [ 9421.212857] Lustre: Failing over lustre-OST0000 [ 9421.263740] Lustre: server umount lustre-OST0000 complete [ 9434.353427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9436.422950] LustreError: 262549:0:(ldlm_lib.c:2930:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9436.426777] LustreError: 262549:0:(ldlm_lib.c:2930:target_recovery_thread()) Skipped 78 previous similar messages [ 9436.607927] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9441.759197] Lustre: *** cfs_fail_loc=715, val=40*** [ 9442.783138] Lustre: *** cfs_fail_loc=715, val=40*** [ 9442.784469] Lustre: Skipped 1 previous similar message [ 9451.999159] Lustre: 3643:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788696892/real 1788696892] req@ffff98768f3cbb80 x1875574859783168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788696908 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 9452.010985] Lustre: 3643:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 17 previous similar messages [ 9452.014427] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [ 9452.017902] Lustre: Skipped 1 previous similar message [ 9452.735336] Lustre: lustre-OST0000: Client 90b009ae-75cc-480a-a0b5-dbdb11b187af (at 192.168.206.40@tcp) reconnected, waiting for 3 clients in recovery for 1:23 [ 9458.143197] Lustre: *** cfs_fail_loc=715, val=40*** [ 9467.359944] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:09 [ 9467.364871] Lustre: Skipped 1 previous similar message [ 9473.503147] Lustre: *** cfs_fail_loc=715, val=40*** [ 9473.506160] Lustre: Skipped 2 previous similar messages [ 9476.471197] LustreError: 262549:0:(ldlm_lib.c:2930:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9476.474809] LustreError: 262549:0:(ldlm_lib.c:2930:target_recovery_thread()) Skipped 78 previous similar messages [ 9478.615132] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9479.182902] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9482.785909] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 08:15:38 (1788696938) [ 9484.240245] Lustre: Failing over lustre-MDT0000 [ 9484.533359] Lustre: server umount lustre-MDT0000 complete [ 9497.829049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9497.909475] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9497.914370] LustreError: Skipped 9 previous similar messages [ 9499.650450] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9503.207898] LustreError: 264096:0:(ldlm_lib.c:2930:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 9506.271123] Lustre: *** cfs_fail_loc=715, val=80*** [ 9506.272528] Lustre: Skipped 2 previous similar messages [ 9516.286171] Lustre: lustre-MDT0000: Client 90b009ae-75cc-480a-a0b5-dbdb11b187af (at 192.168.206.40@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 9516.289904] Lustre: Skipped 1 previous similar message [ 9518.559576] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9522.655138] Lustre: *** cfs_fail_loc=715, val=80*** [ 9522.656974] Lustre: Skipped 1 previous similar message [ 9531.646112] Lustre: lustre-MDT0000: Client 90b009ae-75cc-480a-a0b5-dbdb11b187af (at 192.168.206.40@tcp) reconnected, waiting for 2 clients in recovery for 1:11 [ 9534.943476] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9541.087169] Lustre: *** cfs_fail_loc=715, val=80*** [ 9541.088985] Lustre: Skipped 2 previous similar messages [ 9548.029964] Lustre: lustre-MDT0000: Client 90b009ae-75cc-480a-a0b5-dbdb11b187af (at 192.168.206.40@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 9550.303494] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9566.687736] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9579.774105] Lustre: lustre-MDT0000: Client 90b009ae-75cc-480a-a0b5-dbdb11b187af (at 192.168.206.40@tcp) reconnected, waiting for 2 clients in recovery for 0:23 [ 9579.778539] Lustre: Skipped 1 previous similar message [ 9583.071510] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9583.295986] LustreError: 264096:0:(ldlm_lib.c:2930:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9583.303197] Lustre: 264096:0:(ldlm_lib.c:2976:target_recovery_thread()) too long recovery - read logs [ 9583.303286] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9583.307581] LustreError: dumping log to /tmp/lustre-log.1788697040.264096 [ 9583.314093] Lustre: Skipped 50 previous similar messages [ 9583.365937] Lustre: lustre-MDT0000: Recovery over after 1:24, of 2 clients 2 recovered and 0 were evicted. [ 9583.369590] Lustre: Skipped 16 previous similar messages [ 9583.386380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7831 to 0x280000401:7849) [ 9583.386413] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7758 to 0x2c0000401:7777) [ 9585.439609] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9586.029108] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9589.985716] Lustre: DEBUG MARKER: == replay-single test complete, duration 9304 sec ======== 08:17:26 (1788697046) [ 9590.572393] Lustre: DEBUG MARKER: === replay-single: start cleanup 08:17:26 (1788697046) === [ 9593.315458] Lustre: DEBUG MARKER: === replay-single: finish cleanup 08:17:29 (1788697049) === [ 9624.543554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9624.546637] LustreError: Skipped 10 previous similar messages [ 9624.547819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9624.551700] Lustre: Skipped 47 previous similar messages [ 9630.035194] Lustre: server umount lustre-MDT0000 complete [ 9632.655736] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788697089 with bad export cookie 8072999297525828975 [ 9632.660832] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9632.774555] Lustre: server umount lustre-MDT0001 complete [ 9645.959143] Lustre: server umount lustre-OST0000 complete [ 9659.164322] Lustre: server umount lustre-OST0001 complete [ 9664.414751] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 9665.519322] Key type lgssc unregistered [ 9665.655394] LNet: 267379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9665.657641] LNetError: 267379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9665.665267] LNet: Removed LNI 192.168.206.140@tcp [ 9666.013123] Key type .llcrypt unregistered [ 9666.014186] Key type ._llcrypt unregistered