[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 463544658 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002440] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007294] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010097] LSM: Security Framework initializing [ 0.011042] Yama: becoming mindful. [ 0.012022] SELinux: Initializing. [ 0.014043] *** VALIDATE selinux *** [ 0.020730] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024583] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025112] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027070] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] *** VALIDATE tmpfs *** [ 0.029319] *** VALIDATE proc *** [ 0.031082] *** VALIDATE cgroup *** [ 0.031679] *** VALIDATE cgroup2 *** [ 0.032230] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033118] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036025] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040260] debug: unmapping init [mem 0xffffffffba659000-0xffffffffba660fff] [ 0.042150] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043619] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044019] ... version: 2 [ 0.045010] ... bit width: 48 [ 0.046008] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048008] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.049966] ... event mask: 000000070000000f [ 0.050245] rcu: Hierarchical SRCU implementation. [ 0.052463] smp: Bringing up secondary CPUs ... [ 0.053504] x86: Booting SMP configuration: [ 0.054020] .... node #0, CPUs: #1 #2 #3 [ 0.058591] smp: Brought up 1 node, 4 CPUs [ 0.060012] smpboot: Max logical packages: 1 [ 0.061017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.243890] node 0 deferred pages initialised in 180ms [ 0.246483] devtmpfs: initialized [ 0.247190] x86/mm: Memory block size: 128MB [ 0.249580] gcov: version magic: 0x41383552 [ 0.250611] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.255164] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.258461] pinctrl core: initialized pinctrl subsystem [ 0.260190] [ 0.260839] ************************************************************* [ 0.263016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.266017] ** ** [ 0.268013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.270013] ** ** [ 0.272013] ** This means that this kernel is built to expose internal ** [ 0.274014] ** IOMMU data structures, which may compromise security on ** [ 0.277016] ** your system. ** [ 0.278011] ** ** [ 0.280013] ** If you see this message and you are not debugging the ** [ 0.282012] ** kernel, report this immediately to your vendor! ** [ 0.284010] ** ** [ 0.286011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.288014] ************************************************************* [ 0.289758] NET: Registered protocol family 16 [ 0.291374] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.294063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.296066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.299102] cpuidle: using governor menu [ 0.300652] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.302437] PCI: Using configuration type 1 for base access [ 0.304137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.312110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.313046] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.315112] cryptd: max_cpu_qlen set to 1000 [ 0.317267] ACPI: Added _OSI(Module Device) [ 0.318014] ACPI: Added _OSI(Processor Device) [ 0.319020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.320038] ACPI: Added _OSI(Processor Aggregator Device) [ 0.325209] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.330637] ACPI: Interpreter enabled [ 0.333084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.334023] ACPI: Using IOAPIC for interrupt routing [ 0.336148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.340513] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.350706] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.353057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.356029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.360105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.364048] acpiphp: Slot [2] registered [ 0.365165] acpiphp: Slot [5] registered [ 0.367252] acpiphp: Slot [6] registered [ 0.369154] acpiphp: Slot [7] registered [ 0.370128] acpiphp: Slot [8] registered [ 0.372216] acpiphp: Slot [9] registered [ 0.374144] acpiphp: Slot [10] registered [ 0.376130] acpiphp: Slot [3] registered [ 0.377132] acpiphp: Slot [4] registered [ 0.379135] acpiphp: Slot [11] registered [ 0.380102] acpiphp: Slot [12] registered [ 0.381162] acpiphp: Slot [13] registered [ 0.383120] acpiphp: Slot [14] registered [ 0.385172] acpiphp: Slot [15] registered [ 0.387133] acpiphp: Slot [16] registered [ 0.389129] acpiphp: Slot [17] registered [ 0.390103] acpiphp: Slot [18] registered [ 0.391143] acpiphp: Slot [19] registered [ 0.393163] acpiphp: Slot [20] registered [ 0.395137] acpiphp: Slot [21] registered [ 0.396072] acpiphp: Slot [22] registered [ 0.398177] acpiphp: Slot [23] registered [ 0.399066] acpiphp: Slot [24] registered [ 0.400137] acpiphp: Slot [25] registered [ 0.401128] acpiphp: Slot [26] registered [ 0.403179] acpiphp: Slot [27] registered [ 0.405081] acpiphp: Slot [28] registered [ 0.406093] acpiphp: Slot [29] registered [ 0.407090] acpiphp: Slot [30] registered [ 0.409129] acpiphp: Slot [31] registered [ 0.410070] PCI host bridge to bus 0000:00 [ 0.412023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.414031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.416030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.418025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.420027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.423027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.425224] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.428363] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.431424] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.440019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.445199] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.448021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.451017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.453021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.456757] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.459754] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.461033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.464524] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.470019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.483017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.488018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.493964] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.501023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.509020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.526013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.534846] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.542014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.550019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.570021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.579972] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.587018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.596022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.613014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.626625] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.634018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.642021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.658025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.668363] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.674021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.677025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.700018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.711928] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.718015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.724015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.741015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.749979] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.753411] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.755259] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.757302] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.759120] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.763166] iommu: Default domain type: Passthrough [ 0.764554] SCSI subsystem initialized [ 0.766191] ACPI: bus type USB registered [ 0.767118] usbcore: registered new interface driver usbfs [ 0.769116] usbcore: registered new interface driver hub [ 0.771101] usbcore: registered new device driver usb [ 0.773148] pps_core: LinuxPPS API ver. 1 registered [ 0.775017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.777084] PTP clock support registered [ 0.780043] EDAC MC: Ver: 3.0.0 [ 0.782166] PCI: Using ACPI for IRQ routing [ 0.784359] NetLabel: Initializing [ 0.786082] NetLabel: domain hash size = 128 [ 0.786995] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.789080] NetLabel: unlabeled traffic allowed by default [ 0.791145] vgaarb: loaded [ 0.792304] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.794011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.802241] clocksource: Switched to clocksource kvm-clock [ 0.908932] VFS: Disk quotas dquot_6.6.0 [ 0.910380] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.912496] *** VALIDATE ramfs *** [ 0.913434] *** VALIDATE hugetlbfs *** [ 0.914765] pnp: PnP ACPI init [ 0.917335] pnp: PnP ACPI: found 6 devices [ 0.936424] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.939750] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.942030] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.944511] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.947081] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.950080] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.953093] NET: Registered protocol family 2 [ 0.955891] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.960647] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.963495] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.968595] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.971527] TCP: Hash tables configured (established 65536 bind 65536) [ 0.974349] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.977953] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.980508] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.983249] NET: Registered protocol family 1 [ 0.986536] RPC: Registered named UNIX socket transport module. [ 0.988318] RPC: Registered udp transport module. [ 0.989651] RPC: Registered tcp transport module. [ 0.990954] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.992913] NET: Registered protocol family 44 [ 0.994631] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.997066] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.999346] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.003172] PCI: CLS 0 bytes, default 64 [ 1.005165] Unpacking initramfs... [ 2.351808] debug: unmapping init [mem 0xffff9d7abcc54000-0xffff9d7abffbffff] [ 2.356195] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.358573] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.361368] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.845939] Initialise system trusted keyrings [ 2.847503] Key type blacklist registered [ 2.851916] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.863207] zbud: loaded [ 2.866419] *** VALIDATE nfs *** [ 2.867552] *** VALIDATE nfs4 *** [ 2.868847] pstore: using deflate compression [ 2.871376] Platform Keyring initialized [ 2.960753] NET: Registered protocol family 38 [ 2.962575] Key type asymmetric registered [ 2.964592] Asymmetric key parser 'x509' registered [ 2.966583] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.970192] io scheduler mq-deadline registered [ 2.971508] io scheduler kyber registered [ 2.972833] io scheduler bfq registered [ 2.974353] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.977412] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.980110] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.982976] ACPI: Power Button [PWRF] [ 2.989202] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.997680] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.008475] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.014864] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.026329] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.053125] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.081818] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.088545] Non-volatile memory driver v1.3 [ 3.090449] Linux agpgart interface v0.103 [ 3.120915] virtio_blk virtio1: [vda] 136576 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.123456] vda: detected capacity change from 0 to 69926912 [ 3.137195] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.139716] vdb: detected capacity change from 0 to 1073741824 [ 3.156379] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.159855] vdc: detected capacity change from 0 to 2621440000 [ 3.174811] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.177872] vdd: detected capacity change from 0 to 2621440000 [ 3.190710] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.193487] vde: detected capacity change from 0 to 4294967296 [ 3.208462] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.212194] vdf: detected capacity change from 0 to 4294967296 [ 3.219703] libphy: Fixed MDIO Bus: probed [ 3.227803] usbcore: registered new interface driver usbserial_generic [ 3.230865] usbserial: USB Serial support registered for generic [ 3.234409] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.238713] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.240501] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.242857] mousedev: PS/2 mouse device common for all mice [ 3.246151] rtc_cmos 00:05: RTC can wake from S4 [ 3.248608] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.253101] rtc_cmos 00:05: registered as rtc0 [ 3.254280] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.255783] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.261763] intel_pstate: CPU model not supported [ 3.263153] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.267942] hid: raw HID events driver (C) Jiri Kosina [ 3.271619] usbcore: registered new interface driver usbhid [ 3.273330] usbhid: USB HID core driver [ 3.274662] drop_monitor: Initializing network drop monitor service [ 3.276959] Initializing XFRM netlink socket [ 3.278625] NET: Registered protocol family 10 [ 3.282429] Segment Routing with IPv6 [ 3.283770] NET: Registered protocol family 17 [ 3.285546] mpls_gso: MPLS GSO support [ 3.293951] RAS: Correctable Errors collector initialized. [ 3.295795] AVX version of gcm_enc/dec engaged. [ 3.297162] AES CTR mode by8 optimization enabled [ 3.375671] sched_clock: Marking stable (3375649441, 0)->(4254024927, -878375486) [ 3.379530] registered taskstats version 1 [ 3.381262] Loading compiled-in X.509 certificates [ 3.383846] zswap: loaded using pool lzo/zbud [ 3.411092] Key type big_key registered [ 3.422701] Key type encrypted registered [ 3.424452] ima: No TPM chip found, activating TPM-bypass! [ 3.426607] ima: Allocated hash algorithm: sha1 [ 3.428482] ima: No architecture policies found [ 3.430370] evm: Initialising EVM extended attributes: [ 3.432585] evm: security.selinux [ 3.435360] evm: security.ima [ 3.436617] evm: security.capability [ 3.438008] evm: HMAC attrs: 0x1 [ 3.443830] rtc_cmos 00:05: setting system clock to 2026-06-12 04:25:29 UTC (1781238329) [ 3.449833] debug: unmapping init [mem 0xffffffffbb603000-0xffffffffbb7fffff] [ 3.452710] debug: unmapping init [mem 0xffffffffba382000-0xffffffffba658fff] [ 3.462029] Write protecting the kernel read-only data: 28672k [ 3.465217] debug: unmapping init [mem 0xffffffffb8a03000-0xffffffffb8bfffff] [ 3.467562] debug: unmapping init [mem 0xffffffffb9314000-0xffffffffb93fffff] [ 3.502099] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.510123] systemd[1]: Detected virtualization kvm. [ 3.512024] systemd[1]: Detected architecture x86-64. [ 3.513947] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.540921] systemd[1]: No hostname configured. [ 3.542509] systemd[1]: Set hostname to . [ 3.544323] random: systemd: uninitialized urandom read (16 bytes read) [ 3.546202] systemd[1]: Initializing machine ID from random generator. [ 3.594469] random: ln: uninitialized urandom read (6 bytes read) [ 3.677104] random: systemd: uninitialized urandom read (16 bytes read) [ 3.679940] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.684807] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.689048] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.255683] device-mapper: uevent: version 1.0.3 [ 4.258165] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 4.748881] random: fast init done [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.882649] virtio_net virtio0 ens2: renamed from eth0 [ 4.902166] scsi host0: ata_piix [ 4.924052] scsi host1: ata_piix [ 4.926510] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.929309] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.503659] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.946443] random: crng init done [ 9.947986] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.213965] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.210918] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.451941] SELinux: Disabled at runtime. [ 11.512488] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.519348] systemd[1]: Detected virtualization kvm. [ 11.520945] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.006681] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.010213] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.018258] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.022684] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.025707] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.032458] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.035850] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ 12.176537] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ 12.496121] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.862870] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.889845] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.006797] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.026496] EDAC sbridge: Ver: 1.1.2 [ 14.585787] Key type dns_resolver registered [ 14.882568] NFS: Registering the id_resolver key type [ 14.884506] Key type id_resolver registered [ 14.885830] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg612-server login: [ 45.079722] libcfs: loading out-of-tree module taints kernel. [ 45.118860] Key type ._llcrypt registered [ 45.122992] Key type .llcrypt registered [ 45.189592] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 54.852115] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 55.693775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 55.702983] alg: No test for adler32 (adler32-zlib) [ 56.775641] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 57.187538] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 58.863366] Key type lgssc registered [ 59.658964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 68.948728] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.925795] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 92.584503] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 92.614079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 93.746767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 93.773518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 93.824869] Lustre: lustre-MDT0000: new disk, initializing [ 93.888147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.899172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 95.479861] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.419044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 102.472519] Lustre: 6474:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 102.490919] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 102.494488] Lustre: Skipped 1 previous similar message [ 102.544208] Lustre: lustre-MDT0001: new disk, initializing [ 102.582977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 102.601723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 102.607726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 104.290409] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.961889] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 111.006217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 111.140914] Lustre: lustre-OST0000: new disk, initializing [ 111.143619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 111.147040] Lustre: 8376:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 111.172464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 113.977450] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.230356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 116.238170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 116.267224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 121.256068] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 121.345101] Lustre: lustre-OST0001: new disk, initializing [ 121.346838] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 121.349861] Lustre: 9429:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 121.376433] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 123.718952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 123.725645] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 123.765579] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 124.157068] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.968378] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 137.123604] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 139.557364] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing check_logdir /tmp/testlogs/ [ 141.538775] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing yml_node [ 143.435505] Lustre: DEBUG MARKER: Client: 2.17.53.78 [ 144.512450] Lustre: DEBUG MARKER: MDS: 2.17.53.78 [ 145.595961] Lustre: DEBUG MARKER: OSS: 2.17.53.78 [ 146.259526] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Fri Jun 12 00:27:52 EDT 2026 [ 153.198915] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 153.957744] Lustre: DEBUG MARKER: === replay-single: start setup 00:28:00 (1781238480) === [ 155.342568] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing check_config_client /mnt/lustre [ 163.890306] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 165.535993] Lustre: 13135:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 167.164868] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 169.920496] Lustre: DEBUG MARKER: === replay-single: finish setup 00:28:16 (1781238496) === [ 170.673457] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 00:28:17 (1781238497) [ 174.128278] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 175.010498] Lustre: Failing over lustre-MDT0000 [ 175.072798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 175.073323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 175.079193] Lustre: Skipped 3 previous similar messages [ 175.083365] Lustre: Skipped 3 previous similar messages [ 175.118714] Lustre: server umount lustre-MDT0000 complete [ 184.930029] LustreError: 6482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 184.937023] LustreError: 6482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 190.049654] LustreError: 6480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 190.060781] LustreError: 6480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 190.626598] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 190.628948] LDISKFS-fs (dm-0): recovery complete [ 190.638172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.662690] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 190.772846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.565194] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.168451] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 196.065784] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 196.085512] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 197.645124] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 198.422981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 202.735872] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 00:28:49 (1781238529) [ 203.734728] Lustre: Failing over lustre-OST0000 [ 203.794492] Lustre: server umount lustre-OST0000 complete [ 205.417904] LustreError: 8362:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 205.428090] LustreError: 8362:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 206.304996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 206.305163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 210.529443] LustreError: 15006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 210.537143] LustreError: 15006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 215.648821] LustreError: 9990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 215.667544] LustreError: 9990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 217.137163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 217.243763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.342939] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 218.987453] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 218.989822] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 218.995421] Lustre: Skipped 3 previous similar messages [ 219.429745] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.266331] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 224.007972] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 228.432327] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 00:29:15 (1781238555) [ 231.566601] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 232.538072] Lustre: Failing over lustre-MDT0000 [ 232.636236] Lustre: server umount lustre-MDT0000 complete [ 232.929025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 232.929407] LustreError: 11051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 232.933813] Lustre: Skipped 4 previous similar messages [ 232.941387] LustreError: 11051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 248.194426] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 248.196410] LDISKFS-fs (dm-0): recovery complete [ 248.202346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 248.280377] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 248.377732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.107327] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.164525] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 251.171274] Lustre: lustre-MDT0000: Denying connection for new client 69753c38-23a7-442f-ad14-23d36dfd1422 (at 192.168.206.12@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 253.410186] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 253.412681] Lustre: Skipped 1 previous similar message [ 256.607981] Lustre: lustre-MDT0000: Denying connection for new client 69753c38-23a7-442f-ad14-23d36dfd1422 (at 192.168.206.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 261.729725] Lustre: lustre-MDT0000: Denying connection for new client 69753c38-23a7-442f-ad14-23d36dfd1422 (at 192.168.206.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 266.849533] Lustre: lustre-MDT0000: Denying connection for new client 69753c38-23a7-442f-ad14-23d36dfd1422 (at 192.168.206.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 271.968177] Lustre: lustre-MDT0000: Denying connection for new client 69753c38-23a7-442f-ad14-23d36dfd1422 (at 192.168.206.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 282.210071] Lustre: lustre-MDT0000: Denying connection for new client 69753c38-23a7-442f-ad14-23d36dfd1422 (at 192.168.206.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 282.224020] Lustre: Skipped 1 previous similar message [ 302.688528] Lustre: lustre-MDT0000: Denying connection for new client 69753c38-23a7-442f-ad14-23d36dfd1422 (at 192.168.206.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 302.697978] Lustre: Skipped 3 previous similar messages [ 321.500975] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 321.504134] Lustre: 17780:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 89546657-a428-4705-860d-27d3d1d8c39f@ [ 321.510075] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 321.518166] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 321.518257] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 321.527220] Lustre: Skipped 2 previous similar messages [ 321.537092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 321.537167] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 327.523835] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 00:30:54 (1781238654) [ 330.876232] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 331.741425] Lustre: Failing over lustre-MDT0000 [ 331.744958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [ 331.748594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.755535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.842081] Lustre: server umount lustre-MDT0000 complete [ 335.328186] LustreError: 17779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.335045] LustreError: 17779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 347.235543] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 347.238661] LDISKFS-fs (dm-0): recovery complete [ 347.248135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.309065] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.409829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.034850] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 349.893248] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 349.897064] Lustre: lustre-MDT0000: Denying connection for new client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 349.903790] Lustre: Skipped 3 previous similar messages [ 352.740622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 416.352499] Lustre: lustre-MDT0000: Denying connection for new client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 416.361454] Lustre: Skipped 12 previous similar messages [ 419.500920] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 419.507276] Lustre: 19489:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 69753c38-23a7-442f-ad14-23d36dfd1422@ [ 419.515464] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 419.524114] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 419.524258] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 419.528191] Lustre: Skipped 2 previous similar messages [ 419.546525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 419.546752] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 426.283504] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 00:32:32 (1781238752) [ 429.680154] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 430.706021] Lustre: Failing over lustre-MDT0000 [ 430.827189] Lustre: server umount lustre-MDT0000 complete [ 431.727698] LustreError: 11051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 431.735112] LustreError: 11051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 434.656231] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.662498] Lustre: Skipped 6 previous similar messages [ 446.937054] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 446.939170] LDISKFS-fs (dm-0): recovery complete [ 446.945476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.005952] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 447.073428] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (not set up) [ 447.148827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.492781] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 449.110321] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 452.578445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 452.604946] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 452.624978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 452.624983] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 454.515136] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 455.385288] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 459.753127] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 00:33:06 (1781238786) [ 463.129484] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 463.990155] Lustre: Failing over lustre-MDT0000 [ 464.146655] Lustre: server umount lustre-MDT0000 complete [ 467.936175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.936501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 467.941892] Lustre: Skipped 3 previous similar messages [ 479.508520] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 479.510504] LDISKFS-fs (dm-0): recovery complete [ 479.517081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.591249] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 481.351672] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 483.935714] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 484.834826] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 484.838032] Lustre: Skipped 3 previous similar messages [ 484.866679] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 484.883080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 484.883084] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 486.280298] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 486.944110] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 490.901979] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 00:33:37 (1781238817) [ 493.841593] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 494.649323] Lustre: Failing over lustre-MDT0000 [ 494.827918] Lustre: server umount lustre-MDT0000 complete [ 495.072872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 495.073145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.085013] Lustre: Skipped 3 previous similar messages [ 499.313354] LustreError: 6480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.324020] LustreError: 6480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 33 previous similar messages [ 509.885372] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 509.887173] LDISKFS-fs (dm-0): recovery complete [ 509.891245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.980810] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 511.828951] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 514.656880] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 515.609644] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 515.637968] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 515.638083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:193) [ 517.387546] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 518.202762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 522.892814] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 00:34:09 (1781238849) [ 526.185512] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 526.999327] Lustre: Failing over lustre-MDT0000 [ 527.145720] Lustre: server umount lustre-MDT0000 complete [ 530.915981] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.915983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.926551] Lustre: Skipped 3 previous similar messages [ 542.661557] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 542.663675] LDISKFS-fs (dm-0): recovery complete [ 542.670495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.740680] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 544.528716] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 548.379532] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 548.400948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 548.400953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 550.020622] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 550.857897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 555.114097] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 00:34:41 (1781238881) [ 558.534146] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 559.487655] Lustre: Failing over lustre-MDT0000 [ 559.619408] Lustre: server umount lustre-MDT0000 complete [ 563.679849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 575.187247] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 575.189682] LDISKFS-fs (dm-0): recovery complete [ 575.196040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.275756] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.401518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.410146] Lustre: Skipped 3 previous similar messages [ 576.098256] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 576.103868] Lustre: Skipped 1 previous similar message [ 577.138736] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 580.580748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 580.585358] Lustre: Skipped 11 previous similar messages [ 580.618266] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 580.618630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 582.068295] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 582.744543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 586.724669] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 00:35:13 (1781238913) [ 587.191144] Lustre: *** cfs_fail_loc=13b, val=315*** [ 587.193052] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 587.195064] LustreError: 6483:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b2a9512c0 x1867763809376384/t38654705666(0) o35->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:644/0 lens 392/456 e 0 to 0 dl 1781238924 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 591.658411] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 592.496402] Lustre: Failing over lustre-MDT0000 [ 592.626537] Lustre: server umount lustre-MDT0000 complete [ 595.935679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 595.936504] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.944602] Lustre: Skipped 7 previous similar messages [ 608.123673] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 608.125840] LDISKFS-fs (dm-0): recovery complete [ 608.134158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.340906] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 609.999172] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 613.369344] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 613.373505] Lustre: Skipped 1 previous similar message [ 613.380960] Lustre: 6483:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7a0964fc00 x1867763809376384/t38654705666(0) o35->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:670/0 lens 392/456 e 0 to 0 dl 1781238950 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 613.394893] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 613.395638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 614.836471] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 615.525352] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 619.564700] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 00:35:46 (1781238946) [ 622.814568] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 623.691604] Lustre: Failing over lustre-MDT0000 [ 623.861601] Lustre: server umount lustre-MDT0000 complete [ 628.704578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 628.704952] LustreError: 17779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.718107] LustreError: 17779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 59 previous similar messages [ 639.028312] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 639.030190] LDISKFS-fs (dm-0): recovery complete [ 639.035390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 639.252328] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 641.045356] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 642.656157] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 642.660804] Lustre: Skipped 1 previous similar message [ 644.616159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 644.616159] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 646.042789] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 646.742439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 650.828410] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 00:36:17 (1781238977) [ 653.866744] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 654.251247] Lustre: *** cfs_fail_loc=114, val=0*** [ 655.377932] Lustre: Failing over lustre-MDT0000 [ 655.518583] Lustre: server umount lustre-MDT0000 complete [ 670.643244] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 670.645233] LDISKFS-fs (dm-0): recovery complete [ 670.649204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.691601] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.695977] LustreError: Skipped 2 previous similar messages [ 670.783969] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 672.351864] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 675.834127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 675.834165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 677.474848] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 678.205824] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 682.536201] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 00:36:49 (1781239009) [ 685.624452] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 685.997474] Lustre: *** cfs_fail_loc=128, val=0*** [ 687.153153] Lustre: Failing over lustre-MDT0000 [ 687.297449] Lustre: server umount lustre-MDT0000 complete [ 702.455938] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 702.458057] LDISKFS-fs (dm-0): recovery complete [ 702.463676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 702.648691] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 704.180799] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 708.097764] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 708.098491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 709.643748] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 710.356876] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 714.639555] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 00:37:21 (1781239041) [ 718.222575] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 719.375583] Lustre: Failing over lustre-MDT0000 [ 719.468080] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 719.538561] Lustre: server umount lustre-MDT0000 complete [ 723.424349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 723.428741] LustreError: Skipped 2 previous similar messages [ 735.001274] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 735.003384] LDISKFS-fs (dm-0): recovery complete [ 735.009169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.200618] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 737.248923] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 740.324787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 740.330471] Lustre: Skipped 19 previous similar messages [ 740.466078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 740.466413] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 742.419188] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 743.235796] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 747.816486] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 00:37:54 (1781239074) [ 751.275632] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 752.207508] Lustre: Failing over lustre-MDT0000 [ 752.459712] Lustre: server umount lustre-MDT0000 complete [ 755.680186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 755.687819] Lustre: Skipped 19 previous similar messages [ 768.079170] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 768.082500] LDISKFS-fs (dm-0): recovery complete [ 768.086878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.259877] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 769.772185] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 770.657430] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 770.664504] Lustre: Skipped 3 previous similar messages [ 773.654268] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 773.658698] Lustre: Skipped 4 previous similar messages [ 773.676081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 773.676081] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 775.269700] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 776.067565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 780.262163] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 00:38:26 (1781239106) [ 783.314965] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 786.202569] Lustre: Failing over lustre-MDT0000 [ 786.387605] Lustre: server umount lustre-MDT0000 complete [ 801.256099] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 801.258026] LDISKFS-fs (dm-0): recovery complete [ 801.265414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 801.329569] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 801.333854] LustreError: Skipped 3 previous similar messages [ 801.376644] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (not set up) [ 801.453912] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 803.100178] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 808.386717] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 808.386835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 809.895906] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 810.592206] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 822.957207] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 00:39:09 (1781239149) [ 826.110865] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 826.969283] Lustre: Failing over lustre-MDT0000 [ 827.122484] Lustre: server umount lustre-MDT0000 complete [ 843.157857] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 843.160230] LDISKFS-fs (dm-0): recovery complete [ 843.165091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 843.333794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.338022] Lustre: Skipped 7 previous similar messages [ 843.360833] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 845.063062] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 848.389046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 848.389204] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 849.872512] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 850.636719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 856.643441] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 00:39:43 (1781239183) [ 859.699072] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 860.507571] Lustre: Failing over lustre-MDT0000 [ 860.710697] Lustre: server umount lustre-MDT0000 complete [ 863.711888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 863.718229] LustreError: Skipped 1 previous similar message [ 875.490453] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 875.492302] LDISKFS-fs (dm-0): recovery complete [ 875.497058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.148134] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 881.164594] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 881.170270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 882.667514] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 883.360578] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 887.542090] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 00:40:14 (1781239214) [ 890.831916] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 891.598657] Lustre: Failing over lustre-MDT0000 [ 891.760801] Lustre: server umount lustre-MDT0000 complete [ 894.563524] LustreError: 6481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 894.572689] LustreError: 6481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 118 previous similar messages [ 906.984984] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 906.987260] LDISKFS-fs (dm-0): recovery complete [ 906.991857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.028275] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 912.389329] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 912.389560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 913.930976] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 914.645564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 918.804942] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 00:40:45 (1781239245) [ 921.840886] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 922.677846] Lustre: Failing over lustre-MDT0000 [ 922.814694] Lustre: server umount lustre-MDT0000 complete [ 937.753025] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 937.755035] LDISKFS-fs (dm-0): recovery complete [ 937.761690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.974143] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 937.977301] Lustre: Skipped 2 previous similar messages [ 939.746503] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 943.135895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 943.137363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 944.752989] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 945.465991] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 949.723466] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 00:41:16 (1781239276) [ 953.115816] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 953.969415] Lustre: Failing over lustre-MDT0000 [ 954.112176] LustreError: 49942:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff9d7b2e3e0900) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 956.144825] Lustre: server umount lustre-MDT0000 complete [ 972.190845] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 972.197795] LDISKFS-fs (dm-0): recovery complete [ 972.204970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.411968] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 977.932960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 977.933322] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 979.696539] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 980.474021] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 984.876507] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 00:41:51 (1781239311) [ 988.406591] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 989.264278] Lustre: Failing over lustre-MDT0000 [ 989.456781] Lustre: server umount lustre-MDT0000 complete [ 1004.747199] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1004.749552] LDISKFS-fs (dm-0): recovery complete [ 1004.754935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.637180] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1010.145669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1010.151753] Lustre: Skipped 31 previous similar messages [ 1010.189269] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1010.190031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1011.907791] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1012.709986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1017.280260] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 00:42:23 (1781239343) [ 1021.379958] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1022.461450] Lustre: Failing over lustre-MDT0000 [ 1022.642804] Lustre: server umount lustre-MDT0000 complete [ 1025.503941] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.510178] Lustre: Skipped 30 previous similar messages [ 1038.337943] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1038.340602] LDISKFS-fs (dm-0): recovery complete [ 1038.345399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.916263] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1039.925019] Lustre: Skipped 7 previous similar messages [ 1040.311924] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1043.985311] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1043.989260] Lustre: Skipped 7 previous similar messages [ 1044.004966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1044.005063] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1045.462657] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1046.164952] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1050.487731] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 00:42:57 (1781239377) [ 1054.242541] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1055.101318] Lustre: Failing over lustre-MDT0000 [ 1055.253027] Lustre: server umount lustre-MDT0000 complete [ 1069.996289] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1069.997956] LDISKFS-fs (dm-0): recovery complete [ 1070.003160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1070.058961] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1070.062787] LustreError: Skipped 7 previous similar messages [ 1070.167685] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1070.171218] Lustre: Skipped 3 previous similar messages [ 1071.736779] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1075.210443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1075.211364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1076.483045] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1077.098653] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1080.804352] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 00:43:27 (1781239407) [ 1083.581689] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1084.274671] Lustre: Failing over lustre-MDT0000 [ 1084.448362] Lustre: server umount lustre-MDT0000 complete [ 1099.190800] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1099.192887] LDISKFS-fs (dm-0): recovery complete [ 1099.197488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.922607] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1104.892532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 1104.892542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 1106.340285] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1107.024597] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1111.070506] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 00:43:57 (1781239437) [ 1114.414319] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1115.195917] Lustre: Failing over lustre-MDT0000 [ 1115.354768] Lustre: server umount lustre-MDT0000 complete [ 1120.225203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1120.228625] LustreError: Skipped 7 previous similar messages [ 1130.540425] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1130.542483] LDISKFS-fs (dm-0): recovery complete [ 1130.546883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.330574] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1136.127966] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:897) [ 1136.128218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 1137.618756] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1138.283290] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1142.151607] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 00:44:28 (1781239468) [ 1144.977355] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1145.659210] Lustre: Failing over lustre-MDT0000 [ 1145.791292] Lustre: server umount lustre-MDT0000 complete [ 1160.255429] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1160.257450] LDISKFS-fs (dm-0): recovery complete [ 1160.261791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.873552] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1165.869249] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1165.870138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1167.273025] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1167.916893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1171.706868] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 00:44:58 (1781239498) [ 1174.733577] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1175.486401] Lustre: Failing over lustre-MDT0000 [ 1175.656123] Lustre: server umount lustre-MDT0000 complete [ 1190.320679] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1190.322670] LDISKFS-fs (dm-0): recovery complete [ 1190.327837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1191.978572] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1195.514485] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 1195.514485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 1196.782702] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1197.386763] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1201.035687] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 00:45:27 (1781239527) [ 1203.902552] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1204.581165] Lustre: Failing over lustre-MDT0000 [ 1204.722928] Lustre: server umount lustre-MDT0000 complete [ 1219.519183] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1219.521590] LDISKFS-fs (dm-0): recovery complete [ 1219.527090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1221.160971] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1224.699768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:993) [ 1224.699759] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 1226.208692] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1226.937282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1231.071969] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 00:45:57 (1781239557) [ 1234.156758] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1234.961545] Lustre: Failing over lustre-MDT0000 [ 1235.160565] Lustre: server umount lustre-MDT0000 complete [ 1250.184166] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1250.186646] LDISKFS-fs (dm-0): recovery complete [ 1250.192125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.088655] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1255.432588] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1255.432588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1256.923548] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1257.667569] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1261.427781] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 00:46:28 (1781239588) [ 1264.226422] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1265.030795] Lustre: Failing over lustre-MDT0000 [ 1265.198511] Lustre: server umount lustre-MDT0000 complete [ 1279.712174] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1279.714556] LDISKFS-fs (dm-0): recovery complete [ 1279.721138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.460455] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1285.117792] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1285.117819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1286.473688] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1287.112746] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1291.007135] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 00:46:57 (1781239617) [ 1294.183532] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1295.006337] Lustre: Failing over lustre-MDT0000 [ 1295.104824] Lustre: server umount lustre-MDT0000 complete [ 1309.970498] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1309.972388] LDISKFS-fs (dm-0): recovery complete [ 1309.976578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.832896] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1315.329911] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 1315.330101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 1316.809251] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1317.461620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1321.395088] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 00:47:28 (1781239648) [ 1323.538860] Lustre: 71491:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 2e5707a3-dfce-4882-975a-8dc78cc97ea2 at adminstrative request [ 1327.422181] Lustre: Failing over lustre-MDT0000 [ 1327.571639] Lustre: server umount lustre-MDT0000 complete [ 1341.266883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1341.472077] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1341.475422] Lustre: Skipped 8 previous similar messages [ 1343.053912] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1346.552571] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 1346.552983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1348.165560] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1348.907307] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1351.686520] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1356.064973] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1358.932845] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 00:48:05 (1781239685) [ 1359.408106] Lustre: 73381:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 2e5707a3-dfce-4882-975a-8dc78cc97ea2 at adminstrative request [ 1364.993108] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 00:48:11 (1781239691) [ 1368.262701] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1369.101899] Lustre: Failing over lustre-MDT0000 [ 1369.245130] Lustre: server umount lustre-MDT0000 complete [ 1384.321045] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1384.323309] LDISKFS-fs (dm-0): recovery complete [ 1384.328748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1384.469497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1384.472141] Lustre: Skipped 16 previous similar messages [ 1386.261354] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1389.577357] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 1389.577367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 1391.086168] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1391.782951] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1395.902633] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 00:48:42 (1781239722) [ 1398.931951] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1399.757415] Lustre: Failing over lustre-MDT0000 [ 1399.777850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1399.782078] Lustre: Skipped 3 previous similar messages [ 1399.913159] Lustre: server umount lustre-MDT0000 complete [ 1408.608343] LustreError: 6482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1408.616306] LustreError: 6482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 240 previous similar messages [ 1414.812323] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1414.814021] LDISKFS-fs (dm-0): recovery complete [ 1414.819375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.438386] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1420.280083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1420.280095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1421.528854] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1422.136888] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1425.799961] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 00:49:12 (1781239752) [ 1428.566291] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1429.284103] Lustre: Failing over lustre-MDT0000 [ 1429.434616] Lustre: server umount lustre-MDT0000 complete [ 1444.771435] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1444.773752] LDISKFS-fs (dm-0): recovery complete [ 1444.778692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.550312] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1450.008329] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1450.008335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1451.948782] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1452.747764] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1457.405489] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 00:49:44 (1781239784) [ 1461.030988] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1461.879776] Lustre: Failing over lustre-MDT0000 [ 1462.098636] Lustre: server umount lustre-MDT0000 complete [ 1477.477513] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1477.479307] LDISKFS-fs (dm-0): recovery complete [ 1477.483334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.270513] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1482.749048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1482.749061] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1484.225685] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1484.897850] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1488.895528] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 00:50:15 (1781239815) [ 1492.100249] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1492.828506] Lustre: Failing over lustre-MDT0000 [ 1492.960209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1492.963035] Lustre: Skipped 3 previous similar messages [ 1493.088186] Lustre: server umount lustre-MDT0000 complete [ 1508.704807] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1508.706463] LDISKFS-fs (dm-0): recovery complete [ 1508.711307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.488562] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1513.997160] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 1513.998034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 1515.605816] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1516.346021] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1520.790181] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 00:50:47 (1781239847) [ 1524.178750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1525.217394] Lustre: Failing over lustre-MDT0000 [ 1525.392996] Lustre: server umount lustre-MDT0000 complete [ 1541.269690] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1541.272158] LDISKFS-fs (dm-0): recovery complete [ 1541.277526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.033382] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1546.722799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1546.725622] Lustre: Skipped 67 previous similar messages [ 1546.770592] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1546.770933] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1548.177447] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1548.837472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1552.769301] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 00:51:19 (1781239879) [ 1555.561681] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1556.389702] Lustre: Failing over lustre-MDT0000 [ 1556.543664] Lustre: server umount lustre-MDT0000 complete [ 1556.960263] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.966972] Lustre: Skipped 67 previous similar messages [ 1571.618514] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1571.621337] LDISKFS-fs (dm-0): recovery complete [ 1571.626168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.449993] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1572.453797] Lustre: Skipped 16 previous similar messages [ 1573.319645] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1576.966383] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1576.970103] Lustre: Skipped 16 previous similar messages [ 1576.986429] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1576.986448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1578.272923] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1578.869684] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1582.673649] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 00:51:49 (1781239909) [ 1585.572820] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1586.398566] Lustre: Failing over lustre-MDT0000 [ 1586.577271] Lustre: server umount lustre-MDT0000 complete [ 1601.075970] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1601.078826] LDISKFS-fs (dm-0): recovery complete [ 1601.082817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1601.129970] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.134959] LustreError: Skipped 16 previous similar messages [ 1602.678048] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1606.657865] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1606.657987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1608.147194] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1608.791992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1612.567670] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 00:52:19 (1781239939) [ 1615.700753] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1616.536980] Lustre: Failing over lustre-MDT0000 [ 1616.705480] Lustre: server umount lustre-MDT0000 complete [ 1631.656369] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1631.658023] LDISKFS-fs (dm-0): recovery complete [ 1631.662743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.416255] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1636.859934] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1636.859944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1638.373461] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1639.086321] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1643.244658] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 00:52:49 (1781239969) [ 1646.516800] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1647.379305] Lustre: Failing over lustre-MDT0000 [ 1647.546083] Lustre: server umount lustre-MDT0000 complete [ 1652.192354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1652.195869] LustreError: Skipped 15 previous similar messages [ 1662.137328] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1662.139437] LDISKFS-fs (dm-0): recovery complete [ 1662.144440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1663.728859] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1667.572035] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1667.572079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1668.810928] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1669.411248] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1673.361310] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 00:53:20 (1781240000) [ 1676.449898] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1677.194523] Lustre: Failing over lustre-MDT0000 [ 1677.343195] Lustre: server umount lustre-MDT0000 complete [ 1693.083695] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1693.087118] LDISKFS-fs (dm-0): recovery complete [ 1693.094558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.293086] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1698.820317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 1698.824273] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 1700.260475] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1701.045760] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1705.245392] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 00:53:51 (1781240031) [ 1705.761466] Lustre: 93648:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 2e5707a3-dfce-4882-975a-8dc78cc97ea2 at adminstrative request [ 1711.176482] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 00:53:57 (1781240037) [ 1713.815574] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1714.489277] Lustre: Failing over lustre-MDT0000 [ 1714.667629] Lustre: server umount lustre-MDT0000 complete [ 1719.214092] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1719.215615] LDISKFS-fs (dm-0): recovery complete [ 1719.222053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.385991] Lustre: lustre-MDT0000: Aborting client recovery [ 1719.391461] LustreError: 94928:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1719.399144] Lustre: 94960:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1719.407291] Lustre: 94960:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 2e5707a3-dfce-4882-975a-8dc78cc97ea2@ [ 1719.412349] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1719.418247] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1719.428458] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1719.455974] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 1719.455980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505) [ 1720.997468] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1724.384566] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1733.566800] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 00:54:20 (1781240060) [ 1736.603325] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1737.353720] Lustre: Failing over lustre-MDT0000 [ 1737.531737] Lustre: server umount lustre-MDT0000 complete [ 1742.276373] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1742.280135] LDISKFS-fs (dm-0): recovery complete [ 1742.284739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1742.445810] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1742.474344] Lustre: lustre-MDT0000: Aborting client recovery [ 1742.476897] LustreError: 96771:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1742.480949] Lustre: 96803:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1742.485855] Lustre: 96803:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 1742.489213] Lustre: 96803:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1742.493045] Lustre: 96803:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1742.495930] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1742.499387] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 1742.506428] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 1742.531984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1537) [ 1742.532022] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1537) [ 1744.127908] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1747.936782] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1754.174320] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1754.176279] Lustre: Skipped 1 previous similar message [ 1756.832337] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 00:54:43 (1781240083) [ 1759.916093] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1760.634262] Lustre: Failing over lustre-MDT0000 [ 1760.718286] Lustre: server umount lustre-MDT0000 complete [ 1765.989267] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1765.991118] LDISKFS-fs (dm-0): recovery complete [ 1765.997135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.219036] Lustre: lustre-MDT0000: Aborting client recovery [ 1766.221544] LustreError: 98616:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1766.226159] Lustre: 98649:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1766.229937] Lustre: 98649:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 1766.233022] Lustre: 98649:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1766.238286] Lustre: 98649:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1766.241891] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1766.246086] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 1766.256915] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 1766.283357] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 1766.283681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 1768.166893] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1771.491848] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1780.972540] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 00:55:07 (1781240107) [ 1781.419894] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1781.422508] LustreError: 6481:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b35a83480 x1867763810312576/t201863462916(0) o36->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:328/0 lens 512/456 e 0 to 0 dl 1781240118 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1784.250552] Lustre: Failing over lustre-MDT0000 [ 1784.366869] Lustre: server umount lustre-MDT0000 complete [ 1788.171079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1788.385798] Lustre: lustre-MDT0000: Aborting client recovery [ 1788.388793] LustreError: 99922:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1788.394114] Lustre: 99955:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1788.401422] Lustre: 99955:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 1788.405669] Lustre: 99955:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1788.410309] Lustre: 99955:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1788.414200] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1788.419491] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 1788.428138] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 1788.464536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1601) [ 1788.465959] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 1790.147327] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1793.506705] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1802.778512] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1803.430545] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 00:55:30 (1781240130) [ 1806.683854] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1807.850758] Lustre: Failing over lustre-MDT0000 [ 1808.009596] Lustre: server umount lustre-MDT0000 complete [ 1813.510433] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1813.512656] LDISKFS-fs (dm-0): recovery complete [ 1813.517143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.713652] Lustre: lustre-MDT0000: Aborting client recovery [ 1813.716681] LustreError: 101866:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1813.721083] Lustre: 101899:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1813.724982] Lustre: 101899:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 1813.727766] Lustre: 101899:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1813.734026] Lustre: 101899:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1813.739454] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1813.743818] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 1813.758173] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 1813.791279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1633) [ 1813.794285] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 1815.449570] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1819.109680] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1829.020750] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:55:55 (1781240155) [ 1843.834189] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1844.727056] Lustre: Failing over lustre-MDT0000 [ 1844.917405] Lustre: server umount lustre-MDT0000 complete [ 1859.885677] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1859.887458] LDISKFS-fs (dm-0): recovery complete [ 1859.891295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.083028] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1860.086188] Lustre: Skipped 26 previous similar messages [ 1861.649262] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1865.238091] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 1865.238404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 1866.756312] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1867.455621] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1876.850292] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 00:56:43 (1781240203) [ 1886.429833] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1890.154924] Lustre: Failing over lustre-MDT0000 [ 1890.375194] Lustre: server umount lustre-MDT0000 complete [ 1905.431763] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1905.433733] LDISKFS-fs (dm-0): recovery complete [ 1905.439104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.195063] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1912.086774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 1912.086919] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 1913.572817] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1914.261143] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1924.928835] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 00:57:31 (1781240251) [ 1925.765336] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1926.174021] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1926.177365] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1928.772432] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 00:57:35 (1781240255) [ 1938.562664] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1942.898221] Lustre: Failing over lustre-OST0000 [ 1942.950995] Lustre: server umount lustre-OST0000 complete [ 1958.101927] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 1958.103873] LDISKFS-fs (dm-2): recovery complete [ 1958.107977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1960.451041] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2007.661642] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 00:58:54 (1781240334) [ 2011.100311] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2012.269389] Lustre: Failing over lustre-MDT0000 [ 2012.432854] Lustre: server umount lustre-MDT0000 complete [ 2012.772332] LustreError: 6481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2012.784225] LustreError: 6481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 204 previous similar messages [ 2027.912275] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2027.914307] LDISKFS-fs (dm-0): recovery complete [ 2027.918274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.099841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.103662] Lustre: Skipped 18 previous similar messages [ 2029.882422] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2033.154598] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2033.154624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 2034.691296] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2035.451971] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2049.503066] Lustre: 110958:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781240359/real 1781240359] req@ffff9d7a09049e00 x1867763820633088/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1781240375 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 2049.516506] LustreError: 110958:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2049.517227] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2049.683640] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 00:59:36 (1781240376) [ 2050.528511] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 2051.917851] LustreError: 7965:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2057.183087] LustreError: 7965:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2057.191312] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnecting [ 2057.978788] LustreError: 63324:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2063.327137] LustreError: 63324:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2063.336138] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnecting [ 2064.183158] LustreError: 6481:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2069.471359] LustreError: 6481:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2069.476909] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnecting [ 2070.312770] LustreError: 6481:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2075.615638] LustreError: 6481:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2076.382726] LustreError: 11051:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2081.759099] LustreError: 11051:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2081.762838] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnecting [ 2081.767165] Lustre: Skipped 1 previous similar message [ 2088.573488] LustreError: 6482:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2088.582342] LustreError: 6482:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 2094.047129] LustreError: 6482:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2094.050940] LustreError: 6482:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 2100.191140] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnecting [ 2100.196397] Lustre: Skipped 2 previous similar messages [ 2107.151452] LustreError: 6480:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2107.158928] LustreError: 6480:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 2112.479134] LustreError: 6480:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2112.483261] LustreError: 6480:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 2116.660920] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 01:00:43 (1781240443) [ 2117.620920] LustreError: 11051:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2126.944910] Lustre: lustre-MDT0000: Export ffff9d7b2a981000 already connecting from 192.168.206.12@tcp [ 2132.064097] Lustre: lustre-MDT0000: Export ffff9d7b2a981000 already connecting from 192.168.206.12@tcp [ 2137.184884] Lustre: lustre-MDT0000: Export ffff9d7b2a981000 already connecting from 192.168.206.12@tcp [ 2142.304950] Lustre: lustre-MDT0000: Export ffff9d7b2a981000 already connecting from 192.168.206.12@tcp [ 2142.311899] Lustre: Skipped 1 previous similar message [ 2147.424813] Lustre: lustre-MDT0000: Export ffff9d7b2a981000 already connecting from 192.168.206.12@tcp [ 2157.663116] LustreError: 11051:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2157.663842] Lustre: lustre-MDT0000: Export ffff9d7b2a981000 already connecting from 192.168.206.12@tcp [ 2157.667051] Lustre: 11051:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9d7b263b9e00 x1867763812947840/t0(0) o38->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:0/0 lens 520/416 e 0 to 0 dl 1781240463 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2157.670453] Lustre: Skipped 1 previous similar message [ 2159.734880] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnecting [ 2159.738478] Lustre: Skipped 3 previous similar messages [ 2159.740108] LustreError: 6482:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2185.312380] Lustre: lustre-MDT0000: Export ffff9d7b2a981000 already connecting from 192.168.206.12@tcp [ 2199.785129] LustreError: 6482:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2199.792439] Lustre: 6482:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9d7b417ead00 x1867763812952320/t0(0) o38->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:0/0 lens 520/416 e 0 to 0 dl 1781240505 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2200.672066] LustreError: 7965:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2225.248386] Lustre: lustre-MDT0000: Export ffff9d7b2a981000 already connecting from 192.168.206.12@tcp [ 2225.257587] Lustre: Skipped 2 previous similar messages [ 2240.727120] LustreError: 7965:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2240.732060] Lustre: 7965:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9d7b36e39a40 x1867763812955264/t0(0) o38->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:0/0 lens 520/416 e 0 to 0 dl 1781240546 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2244.789956] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnecting [ 2244.797347] Lustre: Skipped 1 previous similar message [ 2244.800717] LustreError: 6482:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2284.847154] LustreError: 6482:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2284.850348] Lustre: 6482:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9d7b3b79d2c0 x1867763812958848/t0(0) o38->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:0/0 lens 520/416 e 0 to 0 dl 1781240590 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2285.668220] LustreError: 7965:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2310.240173] Lustre: lustre-MDT0000: Export ffff9d7b2a981000 already connecting from 192.168.206.12@tcp [ 2310.244162] Lustre: Skipped 6 previous similar messages [ 2325.719124] LustreError: 7965:0:(ldlm_lib.c:1420:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2325.723723] Lustre: 7965:0:(service.c:2585:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9d7b3ec92580 x1867763812961792/t0(0) o38->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:0/0 lens 520/416 e 0 to 0 dl 1781240631 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2331.887885] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 01:04:18 (1781240658) [ 2334.977535] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2336.569847] Lustre: Failing over lustre-MDT0000 [ 2336.666619] Lustre: server umount lustre-MDT0000 complete [ 2340.319894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2340.324351] LustreError: Skipped 11 previous similar messages [ 2340.327144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2340.333620] Lustre: Skipped 53 previous similar messages [ 2341.632677] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2341.634596] LDISKFS-fs (dm-0): recovery complete [ 2341.639181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.708068] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.711759] LustreError: Skipped 11 previous similar messages [ 2341.775930] Lustre: *** cfs_fail_loc=712, val=0*** [ 2341.778633] LustreError: 9990:0:(service.c:1397:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9d7b3ec903c0 x1867763820779776/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2341.843194] Lustre: lustre-MDT0000: Aborting client recovery [ 2341.845892] LustreError: 115698:0:(ldlm_lib.c:2985:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2341.849096] Lustre: 115731:0:(ldlm_lib.c:2388:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2341.853942] Lustre: 115731:0:(ldlm_lib.c:2388:target_recovery_overseer()) Skipped 2 previous similar messages [ 2341.858317] Lustre: 115731:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2341.863903] Lustre: 115731:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2341.867989] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2341.873172] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2341.882612] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2341.915390] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2341.915394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2343.431578] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2346.979535] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2346.990065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2346.993425] Lustre: Skipped 60 previous similar messages [ 2355.031455] Lustre: Failing over lustre-MDT0000 [ 2355.199777] Lustre: server umount lustre-MDT0000 complete [ 2368.946134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.841914] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2371.681652] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2371.685470] Lustre: Skipped 8 previous similar messages [ 2374.637197] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2374.641530] Lustre: Skipped 8 previous similar messages [ 2374.665871] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2374.667340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2376.394862] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2377.201556] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2381.530719] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 01:05:08 (1781240708) [ 2381.586768] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnecting [ 2381.591219] Lustre: Skipped 2 previous similar messages [ 2385.422093] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 01:05:12 (1781240712) [ 2385.825314] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2385.827729] LustreError: 6489:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b261bc780 x1867763813018624/t0(0) o700->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:177/0 lens 264/248 e 0 to 0 dl 1781240722 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2404.137722] Lustre: Failing over lustre-MDT0000 [ 2404.338820] Lustre: server umount lustre-MDT0000 complete [ 2418.651877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.322905] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2424.311311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2424.311387] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2425.963431] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2426.639899] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2431.349255] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 01:05:57 (1781240757) [ 2432.648259] Lustre: Failing over lustre-OST0000 [ 2432.717109] Lustre: server umount lustre-OST0000 complete [ 2446.436589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2449.015684] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2452.739342] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2453.494196] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2520.375978] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 01:07:26 (1781240846) [ 2524.050442] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2525.176724] Lustre: Failing over lustre-MDT0000 [ 2525.288623] Lustre: server umount lustre-MDT0000 complete [ 2540.901923] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2540.904888] LDISKFS-fs (dm-0): recovery complete [ 2540.909539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.105506] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2541.111043] Lustre: Skipped 9 previous similar messages [ 2542.941790] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2546.360442] Lustre: *** cfs_fail_loc=216, val=0*** [ 2546.362303] LustreError: 121494:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2546.362409] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2547.424327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2609.656509] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 01:08:56 (1781240936) [ 2618.224115] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 01:09:04 (1781240944) [ 2619.459946] Lustre: Failing over lustre-MDT0000 [ 2619.569604] Lustre: server umount lustre-MDT0000 complete [ 2622.944595] LustreError: 63324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2622.957359] LustreError: 63324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 79 previous similar messages [ 2633.609903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.841392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2633.844573] Lustre: Skipped 5 previous similar messages [ 2635.698559] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2639.336460] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2639.340211] LustreError: 123060:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b260c70c0 x1867763813138176/t0(0) o101->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:471/0 lens 328/344 e 0 to 0 dl 1781241016 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2695.273517] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnected, waiting for 2 clients in recovery for 1:06 [ 2695.280172] Lustre: 123060:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2695.302144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 2695.302222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 2696.581583] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2697.079439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2700.764035] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 01:10:27 (1781241027) [ 2702.158693] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2705.142051] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2705.818779] Lustre: Failing over lustre-MDT0000 [ 2705.888530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2705.891053] Lustre: Skipped 3 previous similar messages [ 2706.075383] Lustre: server umount lustre-MDT0000 complete [ 2720.737058] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2720.739954] LDISKFS-fs (dm-0): recovery complete [ 2720.745067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.338055] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2726.406032] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 2726.406361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 2727.823770] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2728.462687] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2732.356283] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 01:10:59 (1781241059) [ 2732.841598] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2737.160827] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2737.855964] Lustre: Failing over lustre-MDT0000 [ 2737.944091] Lustre: server umount lustre-MDT0000 complete [ 2752.631059] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2752.633311] LDISKFS-fs (dm-0): recovery complete [ 2752.637420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.534139] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2758.136298] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 2758.137638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3169) [ 2759.708701] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2760.469708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2765.115282] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 01:11:31 (1781241091) [ 2765.553581] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2769.511763] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2770.229827] Lustre: Failing over lustre-MDT0000 [ 2770.326252] Lustre: server umount lustre-MDT0000 complete [ 2785.236360] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2785.239077] LDISKFS-fs (dm-0): recovery complete [ 2785.244805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2786.878521] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2790.903242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3201) [ 2790.903551] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 2796.077179] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 01:12:02 (1781241122) [ 2797.559435] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2797.562327] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2797.564297] LustreError: 6483:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b4190a940 x1867763813188224/t257698037777(0) o35->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:589/0 lens 392/456 e 0 to 0 dl 1781241134 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2798.690079] Lustre: Failing over lustre-MDT0000 [ 2798.864619] Lustre: server umount lustre-MDT0000 complete [ 2812.158727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.901732] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2817.525280] Lustre: 6484:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7a0a56f840 x1867763813188224/t257698037777(0) o35->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:609/0 lens 392/456 e 0 to 0 dl 1781241154 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2817.531358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3233) [ 2817.531378] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 2819.135261] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2819.850447] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2823.630702] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 01:12:30 (1781241150) [ 2824.007028] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2824.008951] LustreError: 6481:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b35be3840 x1867763813201024/t261993005072(0) o36->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:660/0 lens 504/448 e 0 to 0 dl 1781241205 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2827.722755] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2828.339815] Lustre: Failing over lustre-MDT0000 [ 2828.418826] Lustre: server umount lustre-MDT0000 complete [ 2842.833313] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2842.834680] LDISKFS-fs (dm-0): recovery complete [ 2842.837216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.368496] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2848.238325] Lustre: 6481:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b36e3b480 x1867763813201024/t261993005072(0) o36->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:684/0 lens 504/2880 e 0 to 0 dl 1781241229 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2848.242586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233) [ 2848.243041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 2849.581546] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2850.220608] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2853.802748] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 01:13:00 (1781241180) [ 2854.218362] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2854.220460] LustreError: 6480:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b10f94f00 x1867763813214976/t266287972368(0) o36->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:690/0 lens 504/448 e 0 to 0 dl 1781241235 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2855.530073] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2858.399955] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2859.073335] Lustre: Failing over lustre-MDT0000 [ 2859.173867] Lustre: server umount lustre-MDT0000 complete [ 2873.653789] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2873.656077] LDISKFS-fs (dm-0): recovery complete [ 2873.661358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.317810] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2878.960166] Lustre: 6484:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b260c52c0 x1867763813215744/t266287972369(0) o35->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:671/0 lens 392/456 e 0 to 0 dl 1781241216 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2878.965549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 2878.965559] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3265) [ 2878.969537] Lustre: 6484:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 2883.732287] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 01:13:30 (1781241210) [ 2884.137289] LustreError: 6482:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b4192c780 x1867763813227776/t270582939664(0) o36->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:720/0 lens 504/448 e 0 to 0 dl 1781241265 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2884.148056] LustreError: 6482:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 1 previous similar message [ 2885.451370] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2885.453706] Lustre: Skipped 1 previous similar message [ 2888.657586] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2889.359356] Lustre: Failing over lustre-MDT0000 [ 2889.450985] Lustre: server umount lustre-MDT0000 complete [ 2904.011416] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2904.012804] LDISKFS-fs (dm-0): recovery complete [ 2904.016467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.855210] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2909.177492] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 2909.177774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3329) [ 2909.178870] Lustre: 6481:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b10aecb40 x1867763813227776/t270582939664(0) o36->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:745/0 lens 504/2880 e 0 to 0 dl 1781241290 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2914.125944] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 01:14:00 (1781241240) [ 2914.597738] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2915.952504] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2915.954624] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2915.956764] Lustre: Skipped 2 previous similar messages [ 2920.100560] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2920.891965] Lustre: Failing over lustre-MDT0000 [ 2920.979016] Lustre: server umount lustre-MDT0000 complete [ 2935.414169] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2935.415990] LDISKFS-fs (dm-0): recovery complete [ 2935.420422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2936.945181] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2940.913408] Lustre: 38175:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b3ec903c0 x1867763813239936/t274877906960(0) o35->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:732/0 lens 392/456 e 0 to 0 dl 1781241277 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2940.916287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 2940.916302] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 2945.514337] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 01:14:32 (1781241272) [ 2945.862610] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 2945.865853] LustreError: 11051:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b3e7103c0 x1867763813250304/t279172874255(0) o101->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:26/0 lens 664/608 e 0 to 0 dl 1781241326 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2945.875021] LustreError: 11051:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 1 previous similar message [ 3006.567896] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnecting [ 3006.571843] Lustre: Skipped 4 previous similar messages [ 3006.577923] Lustre: 6481:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b3a445a40 x1867763813250304/t279172874255(0) o101->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:87/0 lens 664/3488 e 0 to 0 dl 1781241387 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 3009.464505] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 01:15:36 (1781241336) [ 3012.804158] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3013.625546] Lustre: Failing over lustre-MDT0000 [ 3013.738689] Lustre: server umount lustre-MDT0000 complete [ 3017.695771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.697119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3017.703702] Lustre: Skipped 57 previous similar messages [ 3017.706422] LustreError: Skipped 9 previous similar messages [ 3029.134768] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3029.137331] LDISKFS-fs (dm-0): recovery complete [ 3029.141517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.195844] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.200925] LustreError: Skipped 12 previous similar messages [ 3030.746496] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3032.161217] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3032.165099] Lustre: Skipped 12 previous similar messages [ 3034.593362] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3034.596260] Lustre: Skipped 54 previous similar messages [ 3034.604865] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 3034.608579] Lustre: Skipped 12 previous similar messages [ 3034.622739] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 3034.622753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3393) [ 3035.933612] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3036.635295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3050.704862] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 01:16:17 (1781241377) [ 3053.851339] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3054.542902] Lustre: Failing over lustre-MDT0000 [ 3054.696329] Lustre: server umount lustre-MDT0000 complete [ 3069.482111] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3069.484014] LDISKFS-fs (dm-0): recovery complete [ 3069.489835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3071.163277] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3075.066851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3425) [ 3075.066862] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 3076.360704] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3076.937773] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3079.326745] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3084.855594] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 01:16:51 (1781241411) [ 3100.128854] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3100.913253] Lustre: Failing over lustre-MDT0000 [ 3101.189944] Lustre: server umount lustre-MDT0000 complete [ 3117.696678] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3117.698844] LDISKFS-fs (dm-0): recovery complete [ 3117.705181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3119.756399] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3123.307148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 3123.309070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 3124.866745] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3125.522152] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3150.438239] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 01:17:57 (1781241477) [ 3155.147838] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3156.099328] Lustre: Failing over lustre-MDT0000 [ 3156.259988] Lustre: server umount lustre-MDT0000 complete [ 3171.141820] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3171.143709] LDISKFS-fs (dm-0): recovery complete [ 3171.148400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.344613] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3171.348387] Lustre: Skipped 12 previous similar messages [ 3172.882692] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3176.452122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 3176.452122] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 3177.864762] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3178.538467] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3182.089737] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3182.845223] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3185.532132] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 01:18:32 (1781241512) [ 3191.549654] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3252.228718] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3252.230210] Lustre: Skipped 1 previous similar message [ 3252.231687] LustreError: 7965:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b10aed2c0 x1867763816040448/t296352743435(0) o36->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:333/0 lens 66040/440 e 0 to 0 dl 1781241633 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3311.722667] Lustre: 6480:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b10f44850 x1867763816040448/t296352743435(0) o36->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:392/0 lens 66040/440 e 0 to 0 dl 1781241692 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3316.801583] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3317.652776] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 01:20:44 (1781241644) [ 3324.488492] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3325.726155] Lustre: Failing over lustre-MDT0000 [ 3325.919143] Lustre: server umount lustre-MDT0000 complete [ 3327.075421] LustreError: 6482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3327.084624] LustreError: 6482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 174 previous similar messages [ 3341.190145] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3341.192269] LDISKFS-fs (dm-0): recovery complete [ 3341.196695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3341.356575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3341.359948] Lustre: Skipped 12 previous similar messages [ 3342.981333] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3346.908979] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 3346.912159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 3348.943432] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3349.812422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3355.611286] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 01:21:22 (1781241682) [ 3367.127051] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3372.076978] Lustre: Failing over lustre-OST0000 [ 3372.129665] Lustre: server umount lustre-OST0000 complete [ 3387.264693] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3387.266927] LDISKFS-fs (dm-2): recovery complete [ 3387.272067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.780310] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3401.354431] Lustre: Failing over lustre-OST0000 [ 3401.409427] Lustre: server umount lustre-OST0000 complete [ 3414.594864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3416.753693] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3420.053592] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3420.680937] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3455.150387] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 01:23:01 (1781241781) [ 3456.232745] Lustre: Failing over lustre-MDT0000 [ 3456.438940] Lustre: server umount lustre-MDT0000 complete [ 3459.041102] LustreError: 41978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781241785 with bad export cookie 10159627119400806507 [ 3469.523807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.981147] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3474.930672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 3474.930695] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 3482.559049] Lustre: Failing over lustre-MDT0000 [ 3482.665499] Lustre: server umount lustre-MDT0000 complete [ 3495.786987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.501037] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3501.057405] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 3501.057420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 3502.579645] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3503.266827] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3507.229766] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 01:23:53 (1781241833) [ 3518.967060] Lustre: Failing over lustre-OST0000 [ 3519.019967] Lustre: server umount lustre-OST0000 complete [ 3532.408525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3534.623291] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3538.122340] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3538.802588] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3543.077497] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 01:24:29 (1781241869) [ 3543.817966] Lustre: Failing over lustre-MDT0000 [ 3544.093225] Lustre: server umount lustre-MDT0000 complete [ 3547.788446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.857265] Lustre: *** cfs_fail_loc=605, val=0*** [ 3547.859633] LustreError: 156770:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0a73e30 failed: rc = -95 [ 3547.864591] LustreError: 156770:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 3547.868191] LustreError: 156770:0:(obd_mount.c:250:lustre_start_simple()) MGS setup error -95 [ 3547.871841] LustreError: 156770:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 3547.875598] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3547.878593] LustreError: 156770:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-MDT0000 [ 3547.888477] Lustre: server umount lustre-MDT0000 complete [ 3547.890805] LustreError: 156770:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3551.121245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3552.737252] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3556.171606] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 01:24:42 (1781241882) [ 3556.354720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3556.354824] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3559.854523] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3561.166695] Lustre: Failing over lustre-MDT0000 [ 3561.432521] Lustre: server umount lustre-MDT0000 complete [ 3576.623120] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3576.625076] LDISKFS-fs (dm-0): recovery complete [ 3576.628729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.954660] Lustre: *** cfs_fail_loc=707, val=0*** [ 3578.410126] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3638.388697] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 3638.408968] Lustre: 158922:0:(ldlm_lib.c:2069:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3638.414043] Lustre: 158922:0:(ldlm_lib.c:2069:extend_recovery_timer()) Skipped 1 previous similar message [ 3638.611779] Lustre: lustre-MDT0000: Recovery over after 1:01, of 2 clients 2 recovered and 0 were evicted. [ 3638.611807] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3638.618464] Lustre: Skipped 10 previous similar messages [ 3638.621117] Lustre: Skipped 40 previous similar messages [ 3638.638342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 3638.638372] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 3640.381110] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3641.073238] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3646.158793] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 01:26:12 (1781241972) [ 3670.883244] LustreError: 7965:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff9d7b10f96d00 x1867763816920832/t0(0) o101->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:707/0 lens 664/0 e 0 to 0 dl 1781242007 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3670.891211] LustreError: 7965:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3681.919120] LustreError: 7965:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3681.928458] LustreError: 6484:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff9d7b2d5692c0 x1867763816921856/t0(0) o35->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:718/0 lens 392/0 e 0 to 0 dl 1781242018 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3687.904896] LustreError: 17127:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff9d7b10f96580 x1867763822415232/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:724/0 lens 544/0 e 0 to 0 dl 1781242024 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3687.913829] LustreError: 17127:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 3694.252754] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 01:27:00 (1781242020) [ 3718.507516] LustreError: 8367:0:(tgt_handler.c:2833:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3729.535131] LustreError: 8367:0:(tgt_handler.c:2833:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3732.905858] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 01:27:39 (1781242059) [ 3756.507617] LustreError: 63324:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff9d7b2d568f00 x1867763816943872/t0(0) o101->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:38/0 lens 576/0 e 0 to 0 dl 1781242093 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3756.516873] LustreError: 63324:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3761.615112] LustreError: 63324:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3761.620929] LustreError: 7965:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff9d7b3edacb40 x1867763816944512/t0(0) o101->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:43/0 lens 664/0 e 0 to 0 dl 1781242098 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3762.596437] LustreError: 6480:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 10000ms [ 3771.874494] LustreError: 41977:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff9d7b3edf9e00 x1867763822457216/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:53/0 lens 544/0 e 0 to 0 dl 1781242108 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3771.885139] LustreError: 41977:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 118 previous similar messages [ 3772.687131] LustreError: 6480:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3785.166472] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 01:28:31 (1781242111) [ 3871.158242] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 01:29:57 (1781242197) [ 3895.026353] LustreError: 6481:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff9d7a07716580 x1867763817020032/t0(0) o101->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:177/0 lens 576/0 e 0 to 0 dl 1781242232 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3895.037201] LustreError: 6481:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 99 previous similar messages [ 3895.041878] LustreError: 6481:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3895.463234] LustreError: 6481:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3911.377604] LustreError: 7965:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3911.382279] LustreError: 7965:0:(service.c:2562:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3911.799122] LustreError: 7965:0:(service.c:2562:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3911.802017] LustreError: 7965:0:(service.c:2562:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3927.338943] LustreError: 7965:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ HIT req@ffff9d7b416ed2c0 x1867763817038976/t0(0) o36->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:209/0 lens 504/0 e 0 to 0 dl 1781242264 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 3927.353195] LustreError: 7965:0:(service.c:2561:ptlrpc_server_handle_request()) Skipped 80 previous similar messages [ 3940.921790] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 01:31:07 (1781242267) [ 3967.321475] Lustre: DEBUG MARKER: phase 2 [ 3971.160801] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 01:31:37 (1781242297) [ 4043.128641] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 01:32:49 (1781242369) [ 4043.799190] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 4044.523766] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 01:32:51 (1781242371) [ 4046.624850] Lustre: DEBUG MARKER: Started rundbench load pid=133572 ... [ 4051.524532] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4053.248323] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 4054.075656] Lustre: Failing over lustre-MDT0000 [ 4054.092515] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 4054.127204] LustreError: 6482:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000128596ae ns: mdt-lustre-MDT0000_UUID lock: ffff9d7b3b430c00/0x8cfe3f097e510897 lrc: 3/0,0 mode: CW/CW res: [0x20001a9e3:0xbe2:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.206.12@tcp remote: 0xb397359ea56e0b53 expref: 4 pid: 6482 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4058.080583] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.082034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4058.087661] Lustre: Skipped 41 previous similar messages [ 4058.092338] Lustre: Skipped 1 previous similar message [ 4059.236333] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 4059.241796] Lustre: Skipped 2 previous similar messages [ 4060.325073] Lustre: server umount lustre-MDT0000 complete [ 4063.201700] LustreError: 17779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4063.219634] LustreError: 17779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 90 previous similar messages [ 4075.644789] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4075.646659] LDISKFS-fs (dm-0): recovery complete [ 4075.654685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.742904] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4075.749042] LustreError: Skipped 8 previous similar messages [ 4075.846834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4075.849742] Lustre: Skipped 7 previous similar messages [ 4075.863137] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4075.866734] Lustre: Skipped 8 previous similar messages [ 4077.419407] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4079.722515] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4079.725703] Lustre: Skipped 11 previous similar messages [ 4081.527060] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5051 to 0x2c0000401:5089) [ 4081.527324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5088 to 0x280000401:5121) [ 4083.031290] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4083.769692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4089.911850] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4091.624335] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 4092.619357] Lustre: Failing over lustre-MDT0001 [ 4092.785802] Lustre: server umount lustre-MDT0001 complete [ 4096.481347] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4096.486784] LustreError: Skipped 16 previous similar messages [ 4108.990390] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 4108.994030] LDISKFS-fs (dm-1): recovery complete [ 4109.004858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4110.829440] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4114.431762] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:449) [ 4114.431848] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) [ 4115.966936] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4116.826778] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4122.695778] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4124.385526] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 4125.229176] Lustre: Failing over lustre-MDT0000 [ 4125.238873] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 4125.416782] Lustre: server umount lustre-MDT0000 complete [ 4140.965170] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4140.967042] LDISKFS-fs (dm-0): recovery complete [ 4140.972715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.779404] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4146.485199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5308 to 0x280000401:5345) [ 4146.487762] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5276 to 0x2c0000401:5313) [ 4148.068315] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4148.907554] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4155.310722] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4157.188042] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 4158.172852] Lustre: Failing over lustre-MDT0001 [ 4158.376297] Lustre: server umount lustre-MDT0001 complete [ 4174.490676] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 4174.493593] LDISKFS-fs (dm-1): recovery complete [ 4174.508269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.422152] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4179.974868] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:420 to 0x280000400:481) [ 4179.975198] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) [ 4181.498835] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4182.202724] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4194.204097] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 01:35:20 (1781242520) [ 4318.243106] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4329.302608] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 4330.257919] Lustre: Failing over lustre-MDT0000 [ 4330.480470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4330.612633] Lustre: server umount lustre-MDT0000 complete [ 4347.003573] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4347.006894] LDISKFS-fs (dm-0): recovery complete [ 4347.014096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.424159] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4352.481740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4352.485544] Lustre: Skipped 14 previous similar messages [ 4354.871402] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4354.875399] Lustre: Skipped 4 previous similar messages [ 4354.894761] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6044 to 0x2c0000401:6081) [ 4354.894944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6077 to 0x280000401:6113) [ 4356.731899] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4357.580481] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4484.098608] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4495.138772] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 4495.997120] Lustre: Failing over lustre-MDT0001 [ 4496.008541] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.12@tcp (stopping) [ 4496.282490] Lustre: server umount lustre-MDT0001 complete [ 4512.829063] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4512.831722] LDISKFS-fs (dm-1): recovery complete [ 4512.839593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4515.118137] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4521.425776] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1445 to 0x2c0000400:1473) [ 4521.425881] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1446 to 0x280000400:1473) [ 4523.590160] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4524.472691] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4579.646763] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 01:41:46 (1781242906) [ 4652.345651] hrtimer: interrupt took 4640426 ns [ 4703.550148] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4714.582985] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 4715.444707] Lustre: Failing over lustre-MDT0000 [ 4715.472735] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 4718.047887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4718.048613] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.052154] LustreError: Skipped 4 previous similar messages [ 4718.058126] Lustre: Skipped 18 previous similar messages [ 4721.233893] Lustre: server umount lustre-MDT0000 complete [ 4722.784368] LustreError: 63324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.793411] LustreError: 63324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 75 previous similar messages [ 4736.614087] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4736.620096] LDISKFS-fs (dm-0): recovery complete [ 4736.627332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4736.704299] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4736.709411] LustreError: Skipped 2 previous similar messages [ 4736.824758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4736.827703] Lustre: Skipped 5 previous similar messages [ 4736.842468] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4736.845443] Lustre: Skipped 5 previous similar messages [ 4738.144882] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4738.148626] Lustre: Skipped 5 previous similar messages [ 4738.629388] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4743.678787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6830 to 0x2c0000401:6849) [ 4743.679030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6862 to 0x280000401:6881) [ 4745.318348] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4746.129845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4870.863629] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4881.739223] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 4882.505490] Lustre: Failing over lustre-MDT0000 [ 4885.472638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4885.475628] Lustre: Skipped 8 previous similar messages [ 4888.150449] Lustre: server umount lustre-MDT0000 complete [ 4902.769059] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4902.770530] LDISKFS-fs (dm-0): recovery complete [ 4902.774074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4904.339844] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4909.108561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7117 to 0x2c0000401:7137) [ 4909.108724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7149 to 0x280000401:7169) [ 4910.594788] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4911.297771] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4915.135255] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 01:47:21 (1781243241) [ 5038.514841] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5049.392558] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 5050.090349] Lustre: Failing over lustre-MDT0001 [ 5050.119667] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5050.122761] Lustre: Skipped 1 previous similar message [ 5050.286795] Lustre: server umount lustre-MDT0001 complete [ 5065.004149] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5065.005773] LDISKFS-fs (dm-1): recovery complete [ 5065.010761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.584245] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5070.305814] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5070.309176] Lustre: Skipped 14 previous similar messages [ 5070.318565] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5070.322952] Lustre: Skipped 3 previous similar messages [ 5070.342709] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2031 to 0x280000400:2049) [ 5070.342735] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2031 to 0x2c0000400:2049) [ 5071.815456] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5072.595750] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5197.331324] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5208.220401] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 5209.012529] Lustre: Failing over lustre-MDT0001 [ 5209.042600] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5209.179637] Lustre: server umount lustre-MDT0001 complete [ 5223.613786] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5223.615882] LDISKFS-fs (dm-1): recovery complete [ 5223.621575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.298265] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5229.049250] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2031 to 0x2c0000400:2081) [ 5229.049344] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2031 to 0x280000400:2081) [ 5230.423909] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5231.060373] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5235.051842] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 01:52:41 (1781243561) [ 5241.383556] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5242.944276] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5243.725956] Lustre: Failing over lustre-OST0000 [ 5243.775505] Lustre: server umount lustre-OST0000 complete [ 5259.567270] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5259.569121] LDISKFS-fs (dm-2): recovery complete [ 5259.575825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5262.501525] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5266.572924] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5267.339194] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5277.252089] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5278.881216] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5279.674239] Lustre: Failing over lustre-OST0000 [ 5279.741580] Lustre: server umount lustre-OST0000 complete [ 5294.690803] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5294.692624] LDISKFS-fs (dm-2): recovery complete [ 5294.698115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5297.182302] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5300.980590] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5301.621700] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5307.600643] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 01:53:54 (1781243634) [ 5430.558249] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5433.181942] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5443.908646] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 5444.581464] Lustre: Failing over lustre-MDT0001 [ 5444.592038] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 5444.594388] LustreError: Skipped 3 previous similar messages [ 5444.595465] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.599022] Lustre: Skipped 14 previous similar messages [ 5444.703177] Lustre: server umount lustre-MDT0001 complete [ 5445.910926] Lustre: Failing over lustre-MDT0000 [ 5448.672192] LustreError: 63324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.677395] LustreError: 63324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 69 previous similar messages [ 5451.880773] Lustre: server umount lustre-MDT0000 complete [ 5466.891735] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5466.895947] LDISKFS-fs (dm-1): recovery complete [ 5466.900360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.020225] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5467.021804] LDISKFS-fs (dm-0): recovery complete [ 5467.025415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.061526] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5467.064874] LustreError: Skipped 1 previous similar message [ 5467.066958] LustreError: 189112:0:(llog.c:1646:llog_backup()) MGC192.168.206.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5467.067153] LustreError: 189113:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGC192.168.206.112@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9d7b3b40c400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5467.070505] Lustre: 189112:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5467.150801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5467.152715] Lustre: Skipped 5 previous similar messages [ 5467.164680] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5467.167551] Lustre: Skipped 5 previous similar messages [ 5468.255864] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5468.258228] Lustre: Skipped 6 previous similar messages [ 5468.516681] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5468.602864] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5469.296516] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2031 to 0x280000400:2113) [ 5469.296526] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2031 to 0x2c0000400:2113) [ 5480.002260] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9978 to 0x2c0000401:10017) [ 5480.002337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10049) [ 5481.609945] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5482.249045] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5482.779328] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5607.067665] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5609.717600] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5620.437752] Lustre: DEBUG MARKER: fail mds2 mds1 2 times [ 5621.114825] Lustre: Failing over lustre-MDT0001 [ 5621.146773] LustreError: 189118:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000ed9e6f54 ns: mdt-lustre-MDT0001_UUID lock: ffff9d7a0f902a00/0x8cfe3f097ea526dc lrc: 3/0,0 mode: PR/PR res: [0x2400036c8:0xfcc:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.206.12@tcp remote: 0xb397359ea580036b expref: 3 pid: 189118 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5621.160255] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.12@tcp (stopping) [ 5621.162683] Lustre: Skipped 5 previous similar messages [ 5621.384617] Lustre: server umount lustre-MDT0001 complete [ 5622.611650] Lustre: Failing over lustre-MDT0000 [ 5622.850662] Lustre: server umount lustre-MDT0000 complete [ 5637.777568] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5637.780189] LDISKFS-fs (dm-1): recovery complete [ 5637.786715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.853748] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5637.855650] LDISKFS-fs (dm-0): recovery complete [ 5637.859868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.899869] LustreError: 192373:0:(llog.c:1646:llog_backup()) MGC192.168.206.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5637.900094] LustreError: 192372:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGC192.168.206.112@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9d7b3f84eb00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5637.902724] Lustre: 192373:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5639.071094] Lustre: 3631:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243949/real 1781243949] req@ffff9d7a10f6de00 x1867763858683264/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243965 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5639.378164] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5639.450856] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5644.151129] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2031 to 0x280000400:2145) [ 5644.151559] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2031 to 0x2c0000400:2145) [ 5644.168060] Lustre: 3633:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243954/real 1781243954] req@ffff9d7a0e6b1a40 x1867763858683904/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5644.176427] Lustre: 3633:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5649.153391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10081) [ 5649.153398] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9978 to 0x2c0000401:10049) [ 5650.443866] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5651.061146] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5651.655927] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5653.855127] Lustre: 3633:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781243959/real 1781243959] req@ffff9d7a10f6f480 x1867763858684416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781243975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5655.207994] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 01:59:41 (1781243981) [ 5657.764208] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5658.689029] Lustre: Failing over lustre-MDT0000 [ 5658.889941] Lustre: server umount lustre-MDT0000 complete [ 5672.929062] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5672.930463] LDISKFS-fs (dm-0): recovery complete [ 5672.933947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.081299] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5674.343425] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5678.049599] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5678.051716] Lustre: Skipped 21 previous similar messages [ 5690.470115] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5690.481253] Lustre: 192378:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b3f92f480 x1867763862141824/t339302445499(339302445499) o101->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:462/0 lens 520/3488 e 0 to 0 dl 1781244027 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5690.512000] Lustre: lustre-MDT0000: Recovery over after 0:16, of 2 clients 2 recovered and 0 were evicted. [ 5690.514731] Lustre: Skipped 7 previous similar messages [ 5690.529727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10113) [ 5690.529954] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10051 to 0x2c0000401:10081) [ 5691.924735] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5692.530683] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5696.114053] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 02:00:22 (1781244022) [ 5698.615464] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5699.517734] Lustre: Failing over lustre-MDT0000 [ 5699.660747] Lustre: server umount lustre-MDT0000 complete [ 5713.715357] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5713.716846] LDISKFS-fs (dm-0): recovery complete [ 5713.719837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5715.050809] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5715.053065] LustreError: 192379:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b261870c0 x1867763862141824/t339302445499(339302445499) o101->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:487/0 lens 520/664 e 0 to 0 dl 1781244052 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5715.090484] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5731.430308] Lustre: lustre-MDT0000: Client 2e5707a3-dfce-4882-975a-8dc78cc97ea2 (at 192.168.206.12@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5731.435453] Lustre: 192378:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b3b59a940 x1867763862141824/t339302445499(339302445499) o101->2e5707a3-dfce-4882-975a-8dc78cc97ea2@192.168.206.12@tcp:503/0 lens 520/3488 e 0 to 0 dl 1781244068 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5731.468285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10145) [ 5731.468285] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10083 to 0x2c0000401:10113) [ 5732.753530] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5733.286084] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5736.916192] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 02:01:03 (1781244063) [ 5738.474483] Lustre: Failing over lustre-OST0000 [ 5738.522739] Lustre: server umount lustre-OST0000 complete [ 5739.862772] Lustre: Failing over lustre-MDT0000 [ 5739.955931] Lustre: server umount lustre-MDT0000 complete [ 5752.956941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.462077] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5755.210142] Lustre: lustre-MDT0000: Denying connection for new client c2bdee0a-3cde-4c0a-aa53-f7a593249efd (at 192.168.206.12@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5758.448685] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10083 to 0x2c0000401:10145) [ 5762.688126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5764.717700] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5768.168354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10177) [ 5768.899992] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 02:01:35 (1781244095) [ 5769.288336] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5769.289818] LustreError: 172162:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7a0f8d2d00 x1867763860039040/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:541/0 lens 1312/4320 e 0 to 0 dl 1781244106 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5771.820170] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5772.461843] Lustre: Failing over lustre-MDT0000 [ 5772.698583] Lustre: server umount lustre-MDT0000 complete [ 5785.567141] Lustre: 192406:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244095/real 1781244095] req@ffff9d7a0f8d1680 x1867763860039040/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1781244111 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5785.579514] Lustre: 192406:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 5786.971515] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5786.973538] LDISKFS-fs (dm-0): recovery complete [ 5786.977555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5788.404890] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5792.244872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10177) [ 5792.244905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10209) [ 5793.418530] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5793.974587] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5797.684159] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 02:02:04 (1781244124) [ 5800.454328] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5802.936892] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5803.534769] Lustre: Failing over lustre-MDT0001 [ 5809.205753] Lustre: server umount lustre-MDT0001 complete [ 5823.283075] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5823.284314] LDISKFS-fs (dm-1): recovery complete [ 5823.288800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.567134] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5828.590171] Lustre: 193130:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b2df4ad00 x1867763874721152/t34359740739(0) o36->c2bdee0a-3cde-4c0a-aa53-f7a593249efd@192.168.206.12@tcp:622/0 lens 560/2880 e 0 to 0 dl 1781244187 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5828.596252] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2156 to 0x2c0000400:2177) [ 5828.596261] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2156 to 0x280000400:2177) [ 5829.666374] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5830.164953] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5833.497569] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 02:02:40 (1781244160) [ 5833.827799] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5833.829812] Lustre: Skipped 1 previous similar message [ 5833.831434] LustreError: 8371:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b26340780 x1867763860101632/t365072220193(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:605/0 lens 2520/4320 e 0 to 0 dl 1781244170 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5833.837482] LustreError: 8371:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 1 previous similar message [ 5836.152640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5838.569420] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5839.149642] Lustre: Failing over lustre-MDT0000 [ 5839.351783] Lustre: server umount lustre-MDT0000 complete [ 5849.055159] Lustre: 202747:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244159/real 1781244159] req@ffff9d7b26341680 x1867763860101632/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1781244175 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5853.148826] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5853.150169] LDISKFS-fs (dm-0): recovery complete [ 5853.152882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5854.533456] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5858.329963] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10209) [ 5858.329972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10241) [ 5859.529572] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5860.077937] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5861.991062] Lustre: Failing over lustre-MDT0001 [ 5862.078231] Lustre: server umount lustre-MDT0001 complete [ 5875.905969] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5875.907202] LDISKFS-fs (dm-1): recovery complete [ 5875.910988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.107657] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5881.329906] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2188 to 0x280000400:2209) [ 5881.329909] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2188 to 0x2c0000400:2209) [ 5882.385680] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5882.901262] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5886.361597] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 02:03:33 (1781244213) [ 5892.021345] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5894.334884] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5894.936785] Lustre: Failing over lustre-MDT0000 [ 5895.199342] Lustre: server umount lustre-MDT0000 complete [ 5896.352637] LustreError: 7413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244222 with bad export cookie 10159627119406494980 [ 5896.354060] Lustre: Failing over lustre-MDT0001 [ 5896.356314] LustreError: 7413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5901.792154] LustreError: 208044:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9d7a0f8d3c00 x1867763860164096/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5901.802295] LustreError: 208044:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5901.899203] Lustre: server umount lustre-MDT0001 complete [ 5916.438347] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5916.440045] LDISKFS-fs (dm-0): recovery complete [ 5916.445340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.448808] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5916.451384] LDISKFS-fs (dm-1): recovery complete [ 5916.457209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.247562] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d7b3ecdcb40 x1867763860164864/t0(0) o250->MGC192.168.206.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5922.836317] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5922.933163] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5926.643363] Lustre: 209160:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b3f7252c0 x1867763874788352/t42949673028(0) o36->c2bdee0a-3cde-4c0a-aa53-f7a593249efd@192.168.206.12@tcp:724/0 lens 560/2880 e 0 to 0 dl 1781244289 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5926.648420] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2220 to 0x280000400:2241) [ 5926.648493] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2220 to 0x2c0000400:2241) [ 5937.791796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10273) [ 5937.791804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10241) [ 5939.139761] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5939.693587] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5940.139441] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5943.623385] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 02:04:30 (1781244270) [ 5943.963794] LustreError: 209899:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b1a4c8b40 x1867763874821248/t47244640324(0) o36->c2bdee0a-3cde-4c0a-aa53-f7a593249efd@192.168.206.12@tcp:716/0 lens 560/448 e 0 to 0 dl 1781244281 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5943.972246] LustreError: 209899:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 1 previous similar message [ 5949.392020] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5950.014927] Lustre: Failing over lustre-MDT0000 [ 5950.242502] Lustre: server umount lustre-MDT0000 complete [ 5960.287680] Lustre: lustre-MDT0001: Client c2bdee0a-3cde-4c0a-aa53-f7a593249efd (at 192.168.206.12@tcp) reconnecting [ 5960.290432] Lustre: Skipped 2 previous similar messages [ 5960.293420] Lustre: 209899:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b35aebc00 x1867763874821248/t47244640324(0) o36->c2bdee0a-3cde-4c0a-aa53-f7a593249efd@192.168.206.12@tcp:732/0 lens 560/2880 e 0 to 0 dl 1781244297 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5964.175024] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5964.176903] LDISKFS-fs (dm-0): recovery complete [ 5964.180124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.503188] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5969.404833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10305) [ 5969.404860] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10273) [ 5970.598958] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5971.215742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5974.917232] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 02:05:01 (1781244301) [ 5975.261282] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5975.262959] Lustre: Skipped 2 previous similar messages [ 5977.628603] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5978.236902] Lustre: Failing over lustre-MDT0001 [ 5978.315668] Lustre: server umount lustre-MDT0001 complete [ 5992.375995] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5992.377563] LDISKFS-fs (dm-1): recovery complete [ 5992.381837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5993.665161] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5997.553112] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2262 to 0x2c0000400:2305) [ 5997.553115] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2262 to 0x280000400:2305) [ 5998.693289] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5999.197149] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6002.626434] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 02:05:29 (1781244329) [ 6008.340613] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6011.006694] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6011.641661] Lustre: Failing over lustre-MDT0000 [ 6011.866801] Lustre: server umount lustre-MDT0000 complete [ 6026.034814] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6026.036556] LDISKFS-fs (dm-0): recovery complete [ 6026.039923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.707228] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6031.361867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10305) [ 6031.361869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10337) [ 6032.700663] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6033.330036] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6035.400235] Lustre: Failing over lustre-MDT0001 [ 6035.480631] Lustre: server umount lustre-MDT0001 complete [ 6049.795716] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6049.797226] LDISKFS-fs (dm-1): recovery complete [ 6049.801461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.219821] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6055.412386] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2316 to 0x2c0000400:2337) [ 6055.412388] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2316 to 0x280000400:2337) [ 6056.620513] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6057.197444] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6060.981349] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 02:06:27 (1781244387) [ 6066.845363] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6069.236087] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6069.847748] Lustre: Failing over lustre-MDT0000 [ 6070.182587] Lustre: server umount lustre-MDT0000 complete [ 6070.752388] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.752924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6070.756650] Lustre: Skipped 54 previous similar messages [ 6070.757031] LustreError: 209160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6070.759774] LustreError: Skipped 13 previous similar messages [ 6070.761581] LustreError: 209160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 161 previous similar messages [ 6071.409647] LustreError: 6467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244397 with bad export cookie 10159627119406508665 [ 6071.410336] Lustre: Failing over lustre-MDT0001 [ 6071.410682] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6071.410686] LustreError: Skipped 9 previous similar messages [ 6071.413787] LustreError: 6467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6071.514844] Lustre: server umount lustre-MDT0001 complete [ 6085.933044] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6085.936269] LDISKFS-fs (dm-0): recovery complete [ 6085.939732] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6085.940982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.941513] LDISKFS-fs (dm-1): recovery complete [ 6085.944536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6086.020476] LustreError: 219732:0:(llog.c:1646:llog_backup()) MGC192.168.206.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6086.023028] Lustre: 219732:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6091.231138] Lustre: 3633:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244401/real 1781244401] req@ffff9d7b35a230c0 x1867763860326144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781244417 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6091.240040] Lustre: 3633:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6096.351313] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d7b3f769e00 x1867763860329088/t0(0) o250->MGC192.168.206.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6096.420412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6096.422448] Lustre: Skipped 17 previous similar messages [ 6096.429580] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6096.432151] Lustre: Skipped 17 previous similar messages [ 6096.992516] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6096.996035] Lustre: Skipped 17 previous similar messages [ 6097.812376] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6097.916368] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6102.574090] Lustre: 219745:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b3ecdda40 x1867763874916864/t55834574916(0) o36->c2bdee0a-3cde-4c0a-aa53-f7a593249efd@192.168.206.12@tcp:119/0 lens 560/2880 e 0 to 0 dl 1781244439 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6102.578509] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2348 to 0x2c0000400:2369) [ 6102.578510] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2348 to 0x280000400:2369) [ 6102.584194] Lustre: 219745:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages [ 6102.616186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10369) [ 6102.616220] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10337) [ 6103.757946] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6104.265404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6104.724168] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6108.169645] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 02:07:14 (1781244434) [ 6108.520738] LustreError: 8370:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b35aea580 x1867763860358400/t386547056654(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:125/0 lens 1488/4320 e 0 to 0 dl 1781244445 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6108.529344] LustreError: 8370:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 3 previous similar messages [ 6111.047433] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6111.666953] Lustre: Failing over lustre-MDT0001 [ 6116.912812] Lustre: server umount lustre-MDT0001 complete [ 6131.005505] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6131.006724] LDISKFS-fs (dm-1): recovery complete [ 6131.010879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6132.462367] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6136.307057] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2380 to 0x2c0000400:2401) [ 6136.307209] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2380 to 0x280000400:2401) [ 6137.498410] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6138.062515] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6141.607681] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 02:07:48 (1781244468) [ 6144.571176] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6145.171728] Lustre: Failing over lustre-MDT0000 [ 6145.390380] Lustre: server umount lustre-MDT0000 complete [ 6158.303078] Lustre: 222036:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244468/real 1781244468] req@ffff9d7b416ed2c0 x1867763860389504/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1984/4320 e 0 to 1 dl 1781244484 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6158.312672] Lustre: 222036:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6159.348979] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6159.350639] LDISKFS-fs (dm-0): recovery complete [ 6159.353344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6160.730818] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6164.988230] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10369) [ 6164.988236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10401) [ 6166.153198] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6166.695427] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6170.045859] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 02:08:16 (1781244496) [ 6172.982553] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6175.485780] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6176.135995] Lustre: Failing over lustre-MDT0000 [ 6176.353922] Lustre: server umount lustre-MDT0000 complete [ 6190.534753] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6190.536092] LDISKFS-fs (dm-0): recovery complete [ 6190.539226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6191.930478] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6195.707582] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10401) [ 6195.707585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10433) [ 6196.857821] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6197.441026] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6199.388693] Lustre: Failing over lustre-MDT0001 [ 6199.392678] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.12@tcp (stopping) [ 6199.394646] Lustre: Skipped 17 previous similar messages [ 6199.467946] Lustre: server umount lustre-MDT0001 complete [ 6213.908565] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6213.910525] LDISKFS-fs (dm-1): recovery complete [ 6213.917039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.322373] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6219.253054] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2380 to 0x280000400:2433) [ 6219.253060] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2380 to 0x2c0000400:2433) [ 6220.358784] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6220.923518] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6224.328664] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 02:09:11 (1781244551) [ 6227.201443] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6229.804027] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6230.455757] Lustre: Failing over lustre-MDT0000 [ 6230.770035] Lustre: server umount lustre-MDT0000 complete [ 6231.980482] LustreError: 41978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244558 with bad export cookie 10159627119406519102 [ 6231.982099] Lustre: Failing over lustre-MDT0001 [ 6231.984405] LustreError: 41978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6238.175730] LustreError: 229205:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9d7b3fe8d2c0 x1867763860472192/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6238.184818] LustreError: 229205:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6238.304687] Lustre: server umount lustre-MDT0001 complete [ 6253.470617] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6253.473822] LDISKFS-fs (dm-0): recovery complete [ 6253.476627] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6253.478966] LDISKFS-fs (dm-1): recovery complete [ 6253.479766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.487631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.574623] LustreError: 230303:0:(llog.c:1646:llog_backup()) MGC192.168.206.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6253.579079] Lustre: 230303:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6257.055856] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d7b1a4c8000 x1867763860473344/t0(0) o250->MGC192.168.206.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6258.624479] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6258.702808] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6262.326730] Lustre: 230311:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9d7b417621c0 x1867763874991872/t68719476738(0) o36->c2bdee0a-3cde-4c0a-aa53-f7a593249efd@192.168.206.12@tcp:305/0 lens 496/2888 e 0 to 0 dl 1781244625 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6262.331688] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2380 to 0x280000400:2465) [ 6262.331789] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2380 to 0x2c0000400:2465) [ 6262.338040] Lustre: 230311:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6262.382270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10465) [ 6262.382271] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10433) [ 6263.591419] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6264.093512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6264.577226] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6268.070511] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 02:09:54 (1781244594) [ 6268.497411] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6268.499670] Lustre: Skipped 6 previous similar messages [ 6271.583601] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6272.285240] Lustre: Failing over lustre-MDT0000 [ 6272.508785] Lustre: server umount lustre-MDT0000 complete [ 6283.871980] Lustre: lustre-MDT0001: Client c2bdee0a-3cde-4c0a-aa53-f7a593249efd (at 192.168.206.12@tcp) reconnecting [ 6283.875397] Lustre: Skipped 1 previous similar message [ 6286.670931] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6286.672088] LDISKFS-fs (dm-0): recovery complete [ 6286.674579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.093084] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6291.938047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6291.941326] Lustre: Skipped 72 previous similar messages [ 6291.956317] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6291.960036] Lustre: Skipped 21 previous similar messages [ 6291.975781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10497) [ 6291.975988] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10465) [ 6293.110373] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6293.634680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6297.163313] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 02:10:23 (1781244623) [ 6300.074308] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6300.815293] Lustre: Failing over lustre-MDT0001 [ 6300.907786] Lustre: server umount lustre-MDT0001 complete [ 6314.927871] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6314.929344] LDISKFS-fs (dm-1): recovery complete [ 6314.934558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.202566] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6320.112063] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2380 to 0x280000400:2497) [ 6320.112068] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2380 to 0x2c0000400:2497) [ 6321.317957] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6321.852888] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6325.345196] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 02:10:52 (1781244652) [ 6328.223334] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6330.721747] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6331.289634] Lustre: Failing over lustre-MDT0000 [ 6331.418542] Lustre: server umount lustre-MDT0000 complete [ 6345.936857] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6345.938377] LDISKFS-fs (dm-0): recovery complete [ 6345.943720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.403798] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6351.366155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10529) [ 6351.366352] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10497) [ 6352.840188] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6353.544197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6355.666075] Lustre: Failing over lustre-MDT0001 [ 6355.761274] Lustre: server umount lustre-MDT0001 complete [ 6370.543177] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6370.544916] LDISKFS-fs (dm-1): recovery complete [ 6370.551033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.993449] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6375.924476] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2380 to 0x280000400:2529) [ 6375.924486] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2380 to 0x2c0000400:2529) [ 6377.045635] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6377.632117] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6381.039215] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 02:11:47 (1781244707) [ 6381.434202] LustreError: 230312:0:(ldlm_lib.c:3327:target_send_reply_msg()) @@@ dropping reply req@ffff9d7b3b726580 x1867763875047168/t81604378626(0) o36->c2bdee0a-3cde-4c0a-aa53-f7a593249efd@192.168.206.12@tcp:398/0 lens 496/456 e 0 to 0 dl 1781244718 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6381.443116] LustreError: 230312:0:(ldlm_lib.c:3327:target_send_reply_msg()) Skipped 6 previous similar messages [ 6383.905548] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6386.346266] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6386.952951] Lustre: Failing over lustre-MDT0000 [ 6387.138042] Lustre: server umount lustre-MDT0000 complete [ 6388.381462] LustreError: 10230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244714 with bad export cookie 10159627119406526669 [ 6388.382482] Lustre: Failing over lustre-MDT0001 [ 6388.385966] LustreError: 10230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6388.482413] Lustre: server umount lustre-MDT0001 complete [ 6403.331543] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6403.333305] LDISKFS-fs (dm-1): recovery complete [ 6403.337757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6403.339250] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6403.343029] LDISKFS-fs (dm-0): recovery complete [ 6403.345779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6403.414323] LustreError: 240905:0:(llog.c:1646:llog_backup()) MGC192.168.206.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6403.417103] Lustre: 240905:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6407.135113] Lustre: 3631:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781244717/real 1781244717] req@ffff9d7a0a8e21c0 x1867763860608384/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781244733 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6407.141973] Lustre: 3631:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6413.279560] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d7b416ed2c0 x1867763860611840/t0(0) o250->MGC192.168.206.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6414.736440] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6414.811393] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6419.507949] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2380 to 0x280000400:2561) [ 6419.507961] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2380 to 0x2c0000400:2561) [ 6419.546400] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10529) [ 6419.546400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10010 to 0x280000401:10561) [ 6420.744656] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6421.267443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6421.736710] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6425.158542] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 02:12:31 (1781244751) [ 6425.782266] Lustre: 242142:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c2bdee0a-3cde-4c0a-aa53-f7a593249efd at adminstrative request [ 6430.480994] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 02:12:37 (1781244757) [ 6432.312560] Lustre: Failing over lustre-MDT0000 [ 6432.492850] Lustre: server umount lustre-MDT0000 complete [ 6445.321342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6446.838446] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6450.698871] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10580 to 0x2c0000401:10625) [ 6450.698870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10613 to 0x280000401:10657) [ 6451.927182] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6452.479500] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6456.088795] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 02:13:02 (1781244782) [ 6460.112173] Lustre: Failing over lustre-OST0000 [ 6460.167725] Lustre: server umount lustre-OST0000 complete [ 6472.774341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6474.510205] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6477.452605] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6477.952563] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6481.412934] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 02:13:28 (1781244808) [ 6482.676324] Lustre: Failing over lustre-MDT0000 [ 6482.887877] Lustre: server umount lustre-MDT0000 complete [ 6485.300925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6486.744100] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6487.440685] Lustre: lustre-MDT0000: Denying connection for new client 8e99a978-8b9b-4e28-a458-8078149a7d64 (at 192.168.206.12@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 6487.446747] Lustre: Skipped 2 previous similar messages [ 6490.617462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10613 to 0x280000401:10689) [ 6490.617479] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10580 to 0x2c0000401:10657) [ 6495.091157] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 02:13:41 (1781244821) [ 6497.821583] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6498.739965] Lustre: Failing over lustre-OST0000 [ 6498.777351] Lustre: server umount lustre-OST0000 complete [ 6513.037886] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6513.039243] LDISKFS-fs (dm-2): recovery complete [ 6513.042514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6515.139231] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6518.313501] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6518.902572] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6523.135426] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 02:14:09 (1781244849) [ 6526.214287] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6528.116225] Lustre: Failing over lustre-OST0000 [ 6528.152165] Lustre: server umount lustre-OST0000 complete [ 6542.391783] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6542.392950] LDISKFS-fs (dm-2): recovery complete [ 6542.395901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6543.989169] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.206.12@tcp inode [0x2000301a1:0x5:0x0] object 0x280000401:10691 extent [0-4194303]: client csum 4e274c0, server csum 3bc2125b [ 6544.400319] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6547.615633] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6548.122721] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6551.354122] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 02:14:38 (1781244878) [ 6553.833288] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6556.209675] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6558.229698] Lustre: Failing over lustre-MDT0000 [ 6558.475228] Lustre: server umount lustre-MDT0000 complete [ 6559.753524] LustreError: 6465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781244885 with bad export cookie 10159627119406552793 [ 6559.755660] Lustre: Failing over lustre-OST0000 [ 6559.758694] LustreError: 6465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6559.792437] Lustre: server umount lustre-OST0000 complete [ 6573.992245] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6573.993495] LDISKFS-fs (dm-0): recovery complete [ 6573.995892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.799677] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9d7b1a76f840 x1867763860761088/t0(0) o250->MGC192.168.206.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6586.153723] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6590.037274] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10580 to 0x2c0000401:10689) [ 6600.271652] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6600.273286] LDISKFS-fs (dm-2): recovery complete [ 6600.276311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6602.056754] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6610.013265] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 02:15:36 (1781244936) [ 6615.242435] Lustre: Failing over lustre-OST0000 [ 6615.299376] Lustre: server umount lustre-OST0000 complete [ 6616.880147] Lustre: Failing over lustre-MDT0000 [ 6617.089080] Lustre: server umount lustre-MDT0000 complete [ 6630.500416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.275877] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6636.025182] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10580 to 0x2c0000401:10721) [ 6639.872794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6641.884536] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6642.607815] Lustre: lustre-OST0000: Denying connection for new client 1fc6ea8d-672e-4888-8aba-5e4e74b7557d (at 192.168.206.12@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 6647.903843] Lustre: lustre-OST0000: Denying connection for new client 1fc6ea8d-672e-4888-8aba-5e4e74b7557d (at 192.168.206.12@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 6658.144066] Lustre: lustre-OST0000: Denying connection for new client 1fc6ea8d-672e-4888-8aba-5e4e74b7557d (at 192.168.206.12@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 6658.151752] Lustre: Skipped 1 previous similar message [ 6678.623914] Lustre: lustre-OST0000: Denying connection for new client 1fc6ea8d-672e-4888-8aba-5e4e74b7557d (at 192.168.206.12@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 6678.632135] Lustre: Skipped 3 previous similar messages [ 6711.500158] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6711.502421] Lustre: 255157:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 8e99a978-8b9b-4e28-a458-8078149a7d64@ [ 6711.506303] Lustre: 255157:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6711.508357] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6711.514893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10740 to 0x280000401:10761) [ 6715.348837] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 6723.308064] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 6725.474592] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 02:17:32 (1781245052) [ 6726.912845] Lustre: Failing over lustre-OST0000 [ 6726.947720] Lustre: server umount lustre-OST0000 complete [ 6727.136171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6727.136201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6727.136589] LustreError: 175638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.136597] LustreError: 175638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 269 previous similar messages [ 6727.138425] LustreError: Skipped 18 previous similar messages [ 6727.151169] Lustre: Skipped 65 previous similar messages [ 6740.034871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6740.084082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6740.086744] Lustre: Skipped 22 previous similar messages [ 6740.089683] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6740.092469] Lustre: Skipped 22 previous similar messages [ 6741.344184] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6741.346809] Lustre: Skipped 21 previous similar messages [ 6741.921891] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6745.999922] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 02:17:52 (1781245072) [ 6747.204631] Lustre: Failing over lustre-OST0000 [ 6747.239568] Lustre: server umount lustre-OST0000 complete [ 6760.111614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6761.190262] LustreError: 258318:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6761.193332] LustreError: 258318:0:(ldlm_lib.c:2886:target_recovery_thread()) Skipped 76 previous similar messages [ 6762.060154] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6767.583354] Lustre: *** cfs_fail_loc=715, val=40*** [ 6776.945719] Lustre: lustre-OST0000: Client 1fc6ea8d-672e-4888-8aba-5e4e74b7557d (at 192.168.206.12@tcp) reconnected, waiting for 3 clients in recovery for 1:24 [ 6777.311185] Lustre: 3630:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781245087/real 1781245087] req@ffff9d7b3ecb4780 x1867763860874368/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781245103 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 6777.321173] Lustre: 3630:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 6783.455170] Lustre: *** cfs_fail_loc=715, val=40*** [ 6783.456593] Lustre: Skipped 2 previous similar messages [ 6793.311976] Lustre: lustre-OST0000: Client 1fc6ea8d-672e-4888-8aba-5e4e74b7557d (at 192.168.206.12@tcp) reconnected, waiting for 3 clients in recovery for 1:08 [ 6793.316990] Lustre: Skipped 2 previous similar messages [ 6799.327157] Lustre: *** cfs_fail_loc=715, val=40*** [ 6799.332746] Lustre: Skipped 2 previous similar messages [ 6801.247163] LustreError: 258318:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6801.251301] LustreError: 258318:0:(ldlm_lib.c:2886:target_recovery_thread()) Skipped 76 previous similar messages [ 6802.725354] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6803.278341] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6806.689498] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 02:18:53 (1781245133) [ 6808.062752] Lustre: Failing over lustre-MDT0000 [ 6808.278555] Lustre: server umount lustre-MDT0000 complete [ 6821.210721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6821.247698] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6821.251813] LustreError: Skipped 10 previous similar messages [ 6822.693163] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6826.469858] LustreError: 259758:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6830.047125] Lustre: *** cfs_fail_loc=715, val=80*** [ 6830.048420] Lustre: Skipped 2 previous similar messages [ 6840.415751] Lustre: lustre-MDT0000: Client 1fc6ea8d-672e-4888-8aba-5e4e74b7557d (at 192.168.206.12@tcp) reconnected, waiting for 2 clients in recovery for 1:26 [ 6840.420044] Lustre: Skipped 2 previous similar messages [ 6842.847437] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6846.431118] Lustre: *** cfs_fail_loc=715, val=80*** [ 6846.432402] Lustre: Skipped 1 previous similar message [ 6855.775772] Lustre: lustre-MDT0000: Client 1fc6ea8d-672e-4888-8aba-5e4e74b7557d (at 192.168.206.12@tcp) reconnected, waiting for 2 clients in recovery for 1:10 [ 6858.207479] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6861.791162] Lustre: *** cfs_fail_loc=715, val=80*** [ 6861.792960] Lustre: Skipped 1 previous similar message [ 6872.159869] Lustre: lustre-MDT0000: Client 1fc6ea8d-672e-4888-8aba-5e4e74b7557d (at 192.168.206.12@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 6874.591521] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6878.175125] Lustre: *** cfs_fail_loc=715, val=80*** [ 6878.176785] Lustre: Skipped 1 previous similar message [ 6888.543877] Lustre: lustre-MDT0000: Client 1fc6ea8d-672e-4888-8aba-5e4e74b7557d (at 192.168.206.12@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [ 6889.951710] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6906.335129] Lustre: 3630:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781245216/real 1781245216] req@ffff9d7b1a76c000 x1867763860941824/t0(0) o400->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 1 dl 1781245232 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 6906.345460] Lustre: 3630:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 6906.349189] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6906.551110] LustreError: 259758:0:(ldlm_lib.c:2886:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6906.557861] Lustre: 259758:0:(ldlm_lib.c:2932:target_recovery_thread()) too long recovery - read logs [ 6906.557942] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6906.560864] LustreError: dumping log to /tmp/lustre-log.1781245232.259758 [ 6906.563269] Lustre: Skipped 50 previous similar messages [ 6906.617903] Lustre: lustre-MDT0000: Recovery over after 1:22, of 2 clients 2 recovered and 0 were evicted. [ 6906.621467] Lustre: Skipped 16 previous similar messages [ 6906.636489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10775 to 0x280000401:10793) [ 6906.638464] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10734 to 0x2c0000401:10753) [ 6907.985535] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6908.509894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6912.210173] Lustre: DEBUG MARKER: == replay-single test complete, duration 6766 sec ======== 02:20:39 (1781245239) [ 6912.724662] Lustre: DEBUG MARKER: === replay-single: start cleanup 02:20:39 (1781245239) === [ 6915.334640] Lustre: DEBUG MARKER: === replay-single: finish cleanup 02:20:42 (1781245242) === [ 6946.273154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6946.278187] Lustre: Skipped 10 previous similar messages [ 6951.704921] Lustre: server umount lustre-MDT0000 complete [ 6954.619620] LustreError: 41978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781245280 with bad export cookie 10159627119406566233 [ 6954.622880] LustreError: 41978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6954.722756] Lustre: server umount lustre-MDT0001 complete [ 6967.045612] Lustre: server umount lustre-OST0000 complete [ 6980.260344] Lustre: server umount lustre-OST0001 complete [ 6986.040922] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 6987.130210] Key type lgssc unregistered [ 6987.259315] LNet: 262921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6987.262103] LNetError: 262921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6987.271407] LNet: Removed LNI 192.168.206.112@tcp [ 6987.639739] Key type .llcrypt unregistered [ 6987.641263] Key type ._llcrypt unregistered