[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 623124840 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002359] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003015] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007011] pid_max: default: 32768 minimum: 301 [ 0.008139] LSM: Security Framework initializing [ 0.009048] Yama: becoming mindful. [ 0.010031] SELinux: Initializing. [ 0.011112] *** VALIDATE selinux *** [ 0.022754] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027739] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031222] *** VALIDATE tmpfs *** [ 0.033286] *** VALIDATE proc *** [ 0.034329] *** VALIDATE cgroup *** [ 0.035000] *** VALIDATE cgroup2 *** [ 0.035253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038022] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042124] debug: unmapping init [mem 0xffffffffa7459000-0xffffffffa7460fff] [ 0.044381] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045809] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047062] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049036] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052028] ... fixed-purpose events: 3 [ 0.053018] ... event mask: 000000070000000f [ 0.054407] rcu: Hierarchical SRCU implementation. [ 0.059064] smp: Bringing up secondary CPUs ... [ 0.061034] x86: Booting SMP configuration: [ 0.062028] .... node #0, CPUs: #1 #2 #3 [ 0.192731] smp: Brought up 1 node, 4 CPUs [ 0.194028] smpboot: Max logical packages: 1 [ 0.195010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.265138] node 0 deferred pages initialised in 53ms [ 0.269252] devtmpfs: initialized [ 0.271271] x86/mm: Memory block size: 128MB [ 0.275207] gcov: version magic: 0x41383552 [ 0.278679] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.281212] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.283599] pinctrl core: initialized pinctrl subsystem [ 0.286440] [ 0.287008] ************************************************************* [ 0.290017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.293017] ** ** [ 0.297028] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.300013] ** ** [ 0.303023] ** This means that this kernel is built to expose internal ** [ 0.306020] ** IOMMU data structures, which may compromise security on ** [ 0.309020] ** your system. ** [ 0.313022] ** ** [ 0.315014] ** If you see this message and you are not debugging the ** [ 0.318017] ** kernel, report this immediately to your vendor! ** [ 0.322021] ** ** [ 0.325021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.328017] ************************************************************* [ 0.331175] NET: Registered protocol family 16 [ 0.332763] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.337049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.340237] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.346139] cpuidle: using governor menu [ 0.348215] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.352731] PCI: Using configuration type 1 for base access [ 0.356130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.368050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.369214] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.375316] cryptd: max_cpu_qlen set to 1000 [ 0.380343] ACPI: Added _OSI(Module Device) [ 0.381000] ACPI: Added _OSI(Processor Device) [ 0.382011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.383097] ACPI: Added _OSI(Processor Aggregator Device) [ 0.388750] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.395448] ACPI: Interpreter enabled [ 0.397066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.400013] ACPI: Using IOAPIC for interrupt routing [ 0.403107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.411156] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.431331] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.434049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.449020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.453172] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.458502] acpiphp: Slot [2] registered [ 0.460879] acpiphp: Slot [5] registered [ 0.463141] acpiphp: Slot [6] registered [ 0.465145] acpiphp: Slot [7] registered [ 0.467101] acpiphp: Slot [8] registered [ 0.469101] acpiphp: Slot [9] registered [ 0.470084] acpiphp: Slot [10] registered [ 0.471090] acpiphp: Slot [3] registered [ 0.473099] acpiphp: Slot [4] registered [ 0.474169] acpiphp: Slot [11] registered [ 0.476425] acpiphp: Slot [12] registered [ 0.477000] acpiphp: Slot [13] registered [ 0.478071] acpiphp: Slot [14] registered [ 0.479095] acpiphp: Slot [15] registered [ 0.481118] acpiphp: Slot [16] registered [ 0.483144] acpiphp: Slot [17] registered [ 0.499554] acpiphp: Slot [18] registered [ 0.504242] acpiphp: Slot [19] registered [ 0.506194] acpiphp: Slot [20] registered [ 0.507162] acpiphp: Slot [21] registered [ 0.509130] acpiphp: Slot [22] registered [ 0.510184] acpiphp: Slot [23] registered [ 0.512119] acpiphp: Slot [24] registered [ 0.513296] acpiphp: Slot [25] registered [ 0.515143] acpiphp: Slot [26] registered [ 0.517109] acpiphp: Slot [27] registered [ 0.518133] acpiphp: Slot [28] registered [ 0.520163] acpiphp: Slot [29] registered [ 0.521142] acpiphp: Slot [30] registered [ 0.524116] acpiphp: Slot [31] registered [ 0.526070] PCI host bridge to bus 0000:00 [ 0.527033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.529025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.531024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.535023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.538027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.541026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.542355] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.544762] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.547077] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.556051] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.561075] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.564024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.565028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.567019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.568592] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.569876] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.572035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.577930] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.581000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.593158] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.602020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.611098] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.642022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.663018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.695013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.714000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.732000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.741000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.772015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.791000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.813016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.824013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.861015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.874396] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.886013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.894018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.925135] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.940729] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.961016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.978019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.998014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.015544] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.022013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.029014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.053020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.068277] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.070377] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.072394] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.073312] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.075702] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.082057] iommu: Default domain type: Passthrough [ 1.084702] SCSI subsystem initialized [ 1.088203] ACPI: bus type USB registered [ 1.090172] usbcore: registered new interface driver usbfs [ 1.092106] usbcore: registered new interface driver hub [ 1.094167] usbcore: registered new device driver usb [ 1.097115] pps_core: LinuxPPS API ver. 1 registered [ 1.101013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.105064] PTP clock support registered [ 1.109073] EDAC MC: Ver: 3.0.0 [ 1.110587] PCI: Using ACPI for IRQ routing [ 1.113322] NetLabel: Initializing [ 1.115011] NetLabel: domain hash size = 128 [ 1.116007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.118123] NetLabel: unlabeled traffic allowed by default [ 1.120109] vgaarb: loaded [ 1.121260] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.124016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.136880] clocksource: Switched to clocksource kvm-clock [ 1.368421] VFS: Disk quotas dquot_6.6.0 [ 1.369753] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.375411] *** VALIDATE ramfs *** [ 1.377844] *** VALIDATE hugetlbfs *** [ 1.382684] pnp: PnP ACPI init [ 1.388374] pnp: PnP ACPI: found 6 devices [ 1.429844] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.432910] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.435253] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.437315] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.439927] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.442164] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.444630] NET: Registered protocol family 2 [ 1.447165] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.452123] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.455887] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.462724] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.467279] TCP: Hash tables configured (established 65536 bind 65536) [ 1.470278] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.473775] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.477534] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.480481] NET: Registered protocol family 1 [ 1.486349] RPC: Registered named UNIX socket transport module. [ 1.488272] RPC: Registered udp transport module. [ 1.491066] RPC: Registered tcp transport module. [ 1.492795] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.495481] NET: Registered protocol family 44 [ 1.497423] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.499509] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.501915] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.505090] PCI: CLS 0 bytes, default 64 [ 1.506883] Unpacking initramfs... [ 4.861575] debug: unmapping init [mem 0xffff94b6fcc54000-0xffff94b6fffbffff] [ 4.878292] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.886553] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.900561] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.629128] Initialise system trusted keyrings [ 7.637154] Key type blacklist registered [ 7.639549] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.676876] zbud: loaded [ 7.679741] *** VALIDATE nfs *** [ 7.685183] *** VALIDATE nfs4 *** [ 7.690238] pstore: using deflate compression [ 7.725876] Platform Keyring initialized [ 8.534901] NET: Registered protocol family 38 [ 8.536448] Key type asymmetric registered [ 8.537784] Asymmetric key parser 'x509' registered [ 8.546881] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 8.563174] io scheduler mq-deadline registered [ 8.564523] io scheduler kyber registered [ 8.565849] io scheduler bfq registered [ 8.576293] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 8.584522] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 8.593676] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 8.605368] ACPI: Power Button [PWRF] [ 8.622229] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 8.648500] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 8.675447] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 8.744861] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.775049] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.819498] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.856520] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.861369] Non-volatile memory driver v1.3 [ 8.862837] Linux agpgart interface v0.103 [ 8.894530] virtio_blk virtio1: [vda] 134160 512-byte logical blocks (68.7 MB/65.5 MiB) [ 8.901300] vda: detected capacity change from 0 to 68689920 [ 8.938567] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.946259] vdb: detected capacity change from 0 to 1073741824 [ 8.987824] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.995611] vdc: detected capacity change from 0 to 2621440000 [ 9.038805] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.044712] vdd: detected capacity change from 0 to 2621440000 [ 9.080558] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.086869] vde: detected capacity change from 0 to 4294967296 [ 9.113675] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 9.120778] vdf: detected capacity change from 0 to 4294967296 [ 9.130878] libphy: Fixed MDIO Bus: probed [ 9.147367] usbcore: registered new interface driver usbserial_generic [ 9.151588] usbserial: USB Serial support registered for generic [ 9.153635] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 9.179373] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 9.181370] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 9.202793] mousedev: PS/2 mouse device common for all mice [ 9.213437] rtc_cmos 00:05: RTC can wake from S4 [ 9.218786] rtc_cmos 00:05: registered as rtc0 [ 9.221811] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 9.225371] intel_pstate: CPU model not supported [ 9.231687] hid: raw HID events driver (C) Jiri Kosina [ 9.234996] usbcore: registered new interface driver usbhid [ 9.236517] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 9.240093] usbhid: USB HID core driver [ 9.240397] drop_monitor: Initializing network drop monitor service [ 9.240532] Initializing XFRM netlink socket [ 9.241305] NET: Registered protocol family 10 [ 9.243869] Segment Routing with IPv6 [ 9.260246] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 9.263229] NET: Registered protocol family 17 [ 9.264246] mpls_gso: MPLS GSO support [ 9.270279] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 9.297250] RAS: Correctable Errors collector initialized. [ 9.303596] AVX version of gcm_enc/dec engaged. [ 9.306731] AES CTR mode by8 optimization enabled [ 9.490561] sched_clock: Marking stable (9490286790, 0)->(11372735177, -1882448387) [ 9.495490] registered taskstats version 1 [ 9.498311] Loading compiled-in X.509 certificates [ 9.499878] zswap: loaded using pool lzo/zbud [ 9.528727] Key type big_key registered [ 9.543982] Key type encrypted registered [ 9.545380] ima: No TPM chip found, activating TPM-bypass! [ 9.547890] ima: Allocated hash algorithm: sha1 [ 9.550428] ima: No architecture policies found [ 9.552668] evm: Initialising EVM extended attributes: [ 9.554820] evm: security.selinux [ 9.556474] evm: security.ima [ 9.557748] evm: security.capability [ 9.559490] evm: HMAC attrs: 0x1 [ 9.562481] rtc_cmos 00:05: setting system clock to 2026-04-11 05:07:30 UTC (1775884050) [ 9.573129] debug: unmapping init [mem 0xffffffffa8403000-0xffffffffa85fffff] [ 9.580457] debug: unmapping init [mem 0xffffffffa7182000-0xffffffffa7458fff] [ 9.595168] Write protecting the kernel read-only data: 28672k [ 9.605701] debug: unmapping init [mem 0xffffffffa5803000-0xffffffffa59fffff] [ 9.612710] debug: unmapping init [mem 0xffffffffa6114000-0xffffffffa61fffff] [ 9.700255] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.714833] systemd[1]: Detected virtualization kvm. [ 9.716841] systemd[1]: Detected architecture x86-64. [ 9.719667] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.768127] systemd[1]: No hostname configured. [ 9.771247] systemd[1]: Set hostname to . [ 9.778106] random: systemd: uninitialized urandom read (16 bytes read) [ 9.783195] systemd[1]: Initializing machine ID from random generator. [ 9.947741] random: ln: uninitialized urandom read (6 bytes read) [ 10.205278] random: systemd: uninitialized urandom read (16 bytes read) [ 10.211149] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 10.248994] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 10.261653] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.996410] device-mapper: uevent: version 1.0.3 [ 12.001439] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 13.869468] virtio_net virtio0 ens2: renamed from eth0 [ 13.970606] random: fast init done [ 14.107279] scsi host0: ata_piix [ 14.141393] scsi host1: ata_piix [ 14.149467] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.160687] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.112473] random: crng init done [ 19.113853] random: 7 urandom warning(s) missed due to ratelimiting [ 20.209841] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 22.688615] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.413696] printk: systemd: 25 output lines suppressed due to ratelimiting [ 27.221141] SELinux: Disabled at runtime. [ 27.361612] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.387036] systemd[1]: Detected virtualization kvm. [ 27.391089] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.303616] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.317392] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.336339] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.349518] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.364110] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.375326] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.394744] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ 29.880318] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 31.195969] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.173605] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.306975] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.006290] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.161817] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit)[ 40.264306] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 41.225447] NFS: Registering the id_resolver key type [ 41.230932] Key type id_resolver registered [ 41.233935] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg430-server login: [ 65.597216] hrtimer: interrupt took 2794255 ns [ 87.277915] libcfs: loading out-of-tree module taints kernel. [ 87.309001] Key type ._llcrypt registered [ 87.311373] Key type .llcrypt registered [ 87.382645] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_hostid [ 100.956723] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 101.900026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 101.913159] alg: No test for adler32 (adler32-zlib) [ 103.124446] Lustre: Lustre: Build Version: 2.17.52_3_g1a00df9 [ 103.751524] LNet: Added LNI 192.168.204.130@tcp [8/256/0/180] [ 105.439239] Key type lgssc registered [ 107.057857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 120.087108] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 151.047268] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing load_modules_local [ 160.546572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 160.561806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.746808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 161.772660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 161.840827] Lustre: lustre-MDT0000: new disk, initializing [ 161.900866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 161.924531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.525254] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.614338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.677404] Lustre: 6500:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.700495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.704827] Lustre: Skipped 1 previous similar message [ 173.758669] Lustre: lustre-MDT0001: new disk, initializing [ 173.804524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.819827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.824665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 176.563532] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.897415] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 186.693663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.872901] Lustre: lustre-OST0000: new disk, initializing [ 186.880855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.936891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.448635] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.999644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.004272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.034833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 200.444647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.529407] Lustre: lustre-OST0001: new disk, initializing [ 200.533345] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 200.577265] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 204.735742] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.981600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 209.995477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 210.031576] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 213.814906] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 216.876128] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 225.853435] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing check_logdir /tmp/testlogs/ [ 229.498680] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing yml_node [ 233.045412] Lustre: DEBUG MARKER: Client: 2.17.52.3 [ 234.923794] Lustre: DEBUG MARKER: MDS: 2.17.52.3 [ 236.657985] Lustre: DEBUG MARKER: OSS: 2.17.52.3 [ 237.753854] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sat Apr 11 01:11:16 EDT 2026 [ 250.799270] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 252.438892] Lustre: DEBUG MARKER: === replay-single: start setup 01:11:31 (1775884291) === [ 255.302265] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing check_config_client /mnt/lustre [ 269.484936] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 272.271470] Lustre: 13168:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 274.922396] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 279.021418] Lustre: DEBUG MARKER: === replay-single: finish setup 01:11:57 (1775884317) === [ 280.548562] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 01:11:59 (1775884319) [ 285.959435] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 287.221034] Lustre: Failing over lustre-MDT0000 [ 287.357093] Lustre: server umount lustre-MDT0000 complete [ 291.808331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 291.809172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 291.831442] Lustre: Skipped 1 previous similar message [ 296.747655] LustreError: 8842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.765032] LustreError: 8842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 301.879391] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 301.888487] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 306.983314] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.010711] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 307.717184] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 307.719264] LDISKFS-fs (dm-0): recovery complete [ 307.727860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 307.763173] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.935244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 311.026202] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 312.111324] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 313.317619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 313.355434] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 316.990325] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 318.368934] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 325.955221] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 01:12:44 (1775884364) [ 327.707325] Lustre: Failing over lustre-OST0000 [ 327.809599] Lustre: server umount lustre-OST0000 complete [ 327.997609] LustreError: 8383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.018942] LustreError: 8383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 328.685827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 328.703388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.718617] Lustre: Skipped 2 previous similar messages [ 333.096025] LustreError: 8386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.113404] LustreError: 8386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 343.339055] LustreError: 8383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.350067] LustreError: 8383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 344.831754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 345.060422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 346.307906] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 346.460193] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 346.460384] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 346.478577] Lustre: Skipped 3 previous similar messages [ 350.134040] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 356.150801] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 357.630514] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 364.835241] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 01:13:23 (1775884403) [ 370.259922] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 371.613716] Lustre: Failing over lustre-MDT0000 [ 371.786485] Lustre: server umount lustre-MDT0000 complete [ 374.753911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 374.766098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.775710] Lustre: Skipped 1 previous similar message [ 374.780767] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.799463] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 390.661093] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 390.664652] LDISKFS-fs (dm-0): recovery complete [ 390.671377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.740288] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 390.960099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.226766] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 395.963143] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 395.970907] Lustre: lustre-MDT0000: Denying connection for new client e90618d0-8c87-497a-a6dc-f858e4171f6b (at 192.168.204.30@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 396.261036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 396.266174] Lustre: Skipped 1 previous similar message [ 401.194522] Lustre: lustre-MDT0000: Denying connection for new client e90618d0-8c87-497a-a6dc-f858e4171f6b (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 401.207696] Lustre: Skipped 1 previous similar message [ 406.312747] Lustre: lustre-MDT0000: Denying connection for new client e90618d0-8c87-497a-a6dc-f858e4171f6b (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 411.431954] Lustre: lustre-MDT0000: Denying connection for new client e90618d0-8c87-497a-a6dc-f858e4171f6b (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 416.554183] Lustre: lustre-MDT0000: Denying connection for new client e90618d0-8c87-497a-a6dc-f858e4171f6b (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 426.794703] Lustre: lustre-MDT0000: Denying connection for new client e90618d0-8c87-497a-a6dc-f858e4171f6b (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 426.812555] Lustre: Skipped 1 previous similar message [ 447.274531] Lustre: lustre-MDT0000: Denying connection for new client e90618d0-8c87-497a-a6dc-f858e4171f6b (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 447.295315] Lustre: Skipped 3 previous similar messages [ 465.500196] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 465.506726] Lustre: 17832:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 63da9d42-2d03-4302-95c1-7fc64a5d34d9@ [ 465.522929] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 465.534065] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 465.534159] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 465.548721] Lustre: Skipped 2 previous similar messages [ 465.562108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 465.562636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 473.870664] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 01:15:12 (1775884512) [ 478.218284] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 479.453969] Lustre: Failing over lustre-MDT0000 [ 479.589492] Lustre: server umount lustre-MDT0000 complete [ 481.250387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 481.256215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.264735] Lustre: Skipped 3 previous similar messages [ 481.268972] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.276870] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 497.154871] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 497.157157] LDISKFS-fs (dm-0): recovery complete [ 497.162096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.224341] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.372387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.936863] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 501.465133] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 501.472343] Lustre: lustre-MDT0000: Denying connection for new client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 501.486316] Lustre: Skipped 3 previous similar messages [ 502.761136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 568.103277] Lustre: lustre-MDT0000: Denying connection for new client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 568.134786] Lustre: Skipped 12 previous similar messages [ 571.503935] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 571.508624] Lustre: 19546:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client e90618d0-8c87-497a-a6dc-f858e4171f6b@ [ 571.516237] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 571.529503] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 571.529649] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 571.539687] Lustre: Skipped 2 previous similar messages [ 571.555733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 571.555833] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 579.878790] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 01:16:58 (1775884618) [ 584.906497] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 586.060886] Lustre: Failing over lustre-MDT0000 [ 586.227488] Lustre: server umount lustre-MDT0000 complete [ 587.232300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 587.240862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.250491] Lustre: Skipped 3 previous similar messages [ 587.253440] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.262287] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 604.574776] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 604.576835] LDISKFS-fs (dm-0): recovery complete [ 604.589655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 604.674949] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 604.822134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 604.844138] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 605.002139] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 607.442833] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 610.283206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 610.330619] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 610.358448] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 610.360754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 612.991640] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 614.090652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 619.705270] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 01:17:38 (1775884658) [ 624.667577] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 626.073593] Lustre: Failing over lustre-MDT0000 [ 626.269850] Lustre: server umount lustre-MDT0000 complete [ 630.754565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 630.755077] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.775242] Lustre: Skipped 6 previous similar messages [ 643.996706] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 643.998892] LDISKFS-fs (dm-0): recovery complete [ 644.006250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.084122] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.265256] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 645.928218] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 646.857796] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 649.709238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 649.715487] Lustre: Skipped 3 previous similar messages [ 649.782817] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 649.816689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 649.816958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 652.138119] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 653.096624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 658.329969] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 01:18:17 (1775884697) [ 663.108978] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 664.485645] Lustre: Failing over lustre-MDT0000 [ 664.678747] Lustre: server umount lustre-MDT0000 complete [ 665.058352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.059335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 665.068587] Lustre: Skipped 2 previous similar messages [ 680.351270] Lustre: 3658:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775884705/real 1775884705] req@ffff94b77dae2300 x1862149492480512/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775884721 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 680.372954] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.177322] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 682.180260] LDISKFS-fs (dm-0): recovery complete [ 682.188531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.655674] LustreError: 3654:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b748521f80 x1862149492488832/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 690.820633] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 691.882313] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 693.336468] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 696.362519] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 696.391909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:193) [ 696.392516] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 698.596182] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 699.542831] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 705.038851] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 01:19:04 (1775884744) [ 709.437482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 710.780039] Lustre: Failing over lustre-MDT0000 [ 710.919058] Lustre: server umount lustre-MDT0000 complete [ 711.648666] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.650090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 711.668276] Lustre: Skipped 2 previous similar messages [ 716.771403] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.791574] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 70 previous similar messages [ 728.035656] Lustre: 3656:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775884752/real 1775884752] req@ffff94b642fdc700 x1862149492507904/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775884768 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 728.049466] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 728.192670] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 728.194917] LDISKFS-fs (dm-0): recovery complete [ 728.201348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.248062] LustreError: 3654:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b64327ea00 x1862149492516480/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 737.456650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 737.461599] Lustre: Skipped 2 previous similar messages [ 737.492829] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 738.087085] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 739.895235] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 742.882371] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 742.886765] Lustre: Skipped 7 previous similar messages [ 742.951831] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 742.976383] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 742.976383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 744.969093] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 745.849078] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 750.722779] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 01:19:50 (1775884790) [ 754.916220] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 756.021104] Lustre: Failing over lustre-MDT0000 [ 756.153852] Lustre: server umount lustre-MDT0000 complete [ 773.547191] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 773.550117] LDISKFS-fs (dm-0): recovery complete [ 773.562244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.649210] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 773.801628] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 776.230790] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 779.316202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 779.316308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 781.628526] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 782.703372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 788.446417] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 01:20:27 (1775884827) [ 789.039646] Lustre: *** cfs_fail_loc=13b, val=315*** [ 789.043772] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 789.049123] LustreError: 6511:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b642041c00 x1862149477632768/t38654705666(0) o35->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:265/0 lens 392/456 e 0 to 0 dl 1775884840 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 794.767482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 795.965346] Lustre: Failing over lustre-MDT0000 [ 796.125858] Lustre: server umount lustre-MDT0000 complete [ 799.713347] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.715859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 799.727293] Lustre: Skipped 6 previous similar messages [ 799.736346] LustreError: Skipped 1 previous similar message [ 813.507146] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 813.509775] LDISKFS-fs (dm-0): recovery complete [ 813.522790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 813.785611] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 814.890584] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 814.896710] Lustre: Skipped 1 previous similar message [ 816.352454] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 819.216275] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 819.222967] Lustre: Skipped 1 previous similar message [ 819.232926] Lustre: 26009:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b642fcfb80 x1862149477632768/t38654705666(0) o35->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:296/0 lens 392/456 e 0 to 0 dl 1775884871 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 819.252619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 819.255814] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 821.500124] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 822.463280] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 827.772530] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 01:21:07 (1775884867) [ 832.603995] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 833.816513] Lustre: Failing over lustre-MDT0000 [ 834.004475] Lustre: server umount lustre-MDT0000 complete [ 850.895915] Lustre: 3655:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775884875/real 1775884875] req@ffff94b74fb6df80 x1862149492589312/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775884891 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 850.909070] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 850.918412] LustreError: Skipped 1 previous similar message [ 851.223992] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 851.227543] LDISKFS-fs (dm-0): recovery complete [ 851.237181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.127462] LustreError: 3654:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b77c38b800 x1862149492597504/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 860.337485] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 862.713994] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 865.845401] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 865.845565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 867.855710] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 868.797525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 874.186854] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 01:21:53 (1775884913) [ 878.883675] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 879.440930] Lustre: *** cfs_fail_loc=114, val=0*** [ 881.004470] Lustre: Failing over lustre-MDT0000 [ 881.123271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 881.125986] Lustre: Skipped 2 previous similar messages [ 881.154442] Lustre: server umount lustre-MDT0000 complete [ 898.470307] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 898.472825] LDISKFS-fs (dm-0): recovery complete [ 898.488579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.707951] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 900.996144] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 904.167877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 904.174284] Lustre: Skipped 15 previous similar messages [ 904.207557] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 904.207785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 906.416195] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 907.407118] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 912.484931] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 01:22:31 (1775884951) [ 917.049229] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 917.745887] Lustre: *** cfs_fail_loc=128, val=0*** [ 919.381406] Lustre: Failing over lustre-MDT0000 [ 919.520709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 919.526876] Lustre: Skipped 3 previous similar messages [ 921.651970] Lustre: server umount lustre-MDT0000 complete [ 938.586871] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 938.589122] LDISKFS-fs (dm-0): recovery complete [ 938.600116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.197213] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 944.147991] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 944.149499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 946.583429] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 947.649192] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 953.374519] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 01:23:12 (1775884992) [ 958.047977] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 959.518584] Lustre: Failing over lustre-MDT0000 [ 959.753386] Lustre: server umount lustre-MDT0000 complete [ 964.575859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.594307] Lustre: Skipped 17 previous similar messages [ 974.821977] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.830102] LustreError: 6507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 139 previous similar messages [ 977.285734] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 977.290306] LDISKFS-fs (dm-0): recovery complete [ 977.300386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.550730] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 977.560274] Lustre: Skipped 1 previous similar message [ 979.944849] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 981.798884] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 981.808368] Lustre: Skipped 3 previous similar messages [ 983.142683] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 983.148421] Lustre: Skipped 3 previous similar messages [ 983.171382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 983.172880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 985.288122] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 986.304994] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 992.010282] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 01:23:51 (1775885031) [ 997.009456] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 998.373428] Lustre: Failing over lustre-MDT0000 [ 998.537718] Lustre: server umount lustre-MDT0000 complete [ 998.879747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 998.890466] LustreError: Skipped 3 previous similar messages [ 1016.092381] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1016.094568] LDISKFS-fs (dm-0): recovery complete [ 1016.102854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.175651] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.181884] LustreError: Skipped 3 previous similar messages [ 1016.314791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.321105] Lustre: Skipped 6 previous similar messages [ 1018.933573] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1021.528690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1021.528691] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1024.023700] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1025.036112] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1030.559712] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 01:24:29 (1775885069) [ 1034.723212] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1040.560884] Lustre: Failing over lustre-MDT0000 [ 1040.715799] Lustre: server umount lustre-MDT0000 complete [ 1057.248238] Lustre: 3657:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775885082/real 1775885082] req@ffff94b74870c700 x1862149492729600/t0(0) o400->MGC192.168.204.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775885098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1058.152109] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1058.155163] LDISKFS-fs (dm-0): recovery complete [ 1058.166058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.492857] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff426f1a0964d14b [ 1070.144885] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1075.384661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1075.384862] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1077.657662] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1078.568289] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1096.245986] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 01:25:35 (1775885135) [ 1100.796528] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1102.091234] Lustre: Failing over lustre-MDT0000 [ 1102.251870] Lustre: server umount lustre-MDT0000 complete [ 1119.485116] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1119.487254] LDISKFS-fs (dm-0): recovery complete [ 1119.493787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1119.760685] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1119.764360] Lustre: Skipped 2 previous similar messages [ 1122.133672] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1124.919143] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1124.919908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1127.097790] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1128.119841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1135.719691] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 01:26:14 (1775885174) [ 1140.391572] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1141.507393] Lustre: Failing over lustre-MDT0000 [ 1141.697343] Lustre: server umount lustre-MDT0000 complete [ 1158.566953] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1158.569176] LDISKFS-fs (dm-0): recovery complete [ 1158.574740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1161.245812] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1164.259069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1164.263889] Lustre: Skipped 24 previous similar messages [ 1164.333248] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1164.333600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1166.401226] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1167.445361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1172.865656] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 01:26:52 (1775885212) [ 1177.270700] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1178.535960] Lustre: Failing over lustre-MDT0000 [ 1178.719837] Lustre: server umount lustre-MDT0000 complete [ 1195.449543] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1195.451637] LDISKFS-fs (dm-0): recovery complete [ 1195.459622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1197.954865] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1201.171679] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1201.174411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1203.238064] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1204.171580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1209.524763] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 01:27:28 (1775885248) [ 1213.692471] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1215.010071] Lustre: Failing over lustre-MDT0000 [ 1215.162269] Lustre: server umount lustre-MDT0000 complete [ 1232.157945] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1232.160785] LDISKFS-fs (dm-0): recovery complete [ 1232.170526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.580577] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1237.613375] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1237.619846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1240.154496] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1241.316135] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1246.696512] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 01:28:05 (1775885285) [ 1251.208484] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1252.313384] Lustre: Failing over lustre-MDT0000 [ 1252.437159] Lustre: server umount lustre-MDT0000 complete [ 1252.833218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.847556] Lustre: Skipped 27 previous similar messages [ 1268.574852] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1268.577105] LDISKFS-fs (dm-0): recovery complete [ 1268.582344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1269.031942] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1269.035233] Lustre: Skipped 6 previous similar messages [ 1270.472855] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1273.859213] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1273.864406] Lustre: Skipped 6 previous similar messages [ 1273.884355] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1273.884750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1275.875162] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1276.821960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1281.937110] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 01:28:41 (1775885321) [ 1286.045992] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1287.067602] Lustre: Failing over lustre-MDT0000 [ 1287.235958] Lustre: server umount lustre-MDT0000 complete [ 1289.184021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1289.188655] LustreError: Skipped 4 previous similar messages [ 1303.753910] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1303.755569] LDISKFS-fs (dm-0): recovery complete [ 1303.760516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1303.825238] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1303.832958] LustreError: Skipped 6 previous similar messages [ 1305.794696] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1309.208897] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1309.209848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1311.277541] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1312.129915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1317.000816] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 01:29:16 (1775885356) [ 1321.374889] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1322.584142] Lustre: Failing over lustre-MDT0000 [ 1322.789867] Lustre: server umount lustre-MDT0000 complete [ 1339.706909] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1339.712491] LDISKFS-fs (dm-0): recovery complete [ 1339.724689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.871912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1339.876991] Lustre: Skipped 3 previous similar messages [ 1342.228731] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1345.074828] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1345.075247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1347.010288] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1347.915409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1353.032660] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 01:29:52 (1775885392) [ 1357.206767] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1358.150102] Lustre: Failing over lustre-MDT0000 [ 1358.348909] Lustre: server umount lustre-MDT0000 complete [ 1374.263221] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1374.265489] LDISKFS-fs (dm-0): recovery complete [ 1374.269366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.243150] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1379.873393] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1379.873748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1381.491690] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1382.270677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1387.002645] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 01:30:26 (1775885426) [ 1390.565996] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1391.467871] Lustre: Failing over lustre-MDT0000 [ 1391.614445] Lustre: server umount lustre-MDT0000 complete [ 1407.717658] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1407.719512] LDISKFS-fs (dm-0): recovery complete [ 1407.724480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.902671] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1407.908783] Lustre: Skipped 7 previous similar messages [ 1410.054466] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1413.137344] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 1413.137403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 1415.182178] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1416.074025] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1421.743145] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 01:31:00 (1775885460) [ 1426.073144] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1427.135971] Lustre: Failing over lustre-MDT0000 [ 1427.352485] Lustre: server umount lustre-MDT0000 complete [ 1444.235260] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1444.238112] LDISKFS-fs (dm-0): recovery complete [ 1444.247521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.810825] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1450.002689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:897) [ 1450.002957] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 1451.808954] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1452.719815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1457.554724] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 01:31:37 (1775885497) [ 1461.615976] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1462.711043] Lustre: Failing over lustre-MDT0000 [ 1462.896823] Lustre: server umount lustre-MDT0000 complete [ 1479.339187] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1479.342264] LDISKFS-fs (dm-0): recovery complete [ 1479.349169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.489910] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1484.854934] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1484.859955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1486.889535] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1487.881878] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1492.451994] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 01:32:11 (1775885531) [ 1496.155200] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1497.062806] Lustre: Failing over lustre-MDT0000 [ 1497.300378] Lustre: server umount lustre-MDT0000 complete [ 1500.129707] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1500.140097] LustreError: 6508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 238 previous similar messages [ 1513.205334] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1513.207316] LDISKFS-fs (dm-0): recovery complete [ 1513.213517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.130865] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1518.607152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 1518.608283] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 1520.278760] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1521.093718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1525.615317] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 01:32:45 (1775885565) [ 1529.561292] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1530.676296] Lustre: Failing over lustre-MDT0000 [ 1530.860692] Lustre: server umount lustre-MDT0000 complete [ 1547.220735] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1547.223116] LDISKFS-fs (dm-0): recovery complete [ 1547.232356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.409912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.413362] Lustre: Skipped 13 previous similar messages [ 1549.284979] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1552.898896] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 1552.898970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:993) [ 1554.437376] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1555.127448] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1559.352429] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 01:33:18 (1775885598) [ 1562.699534] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1563.597776] Lustre: Failing over lustre-MDT0000 [ 1563.762756] Lustre: server umount lustre-MDT0000 complete [ 1578.639570] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1578.641039] LDISKFS-fs (dm-0): recovery complete [ 1578.644853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1580.301965] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1584.149613] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1584.149671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1585.673902] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1586.448559] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1590.473989] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 01:33:50 (1775885630) [ 1593.434864] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1594.252996] Lustre: Failing over lustre-MDT0000 [ 1594.336807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1594.339625] Lustre: Skipped 3 previous similar messages [ 1594.500251] Lustre: server umount lustre-MDT0000 complete [ 1609.434291] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1609.436279] LDISKFS-fs (dm-0): recovery complete [ 1609.440505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.249697] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1614.854409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1614.854591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1616.291091] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1616.976781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1621.102278] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 01:34:20 (1775885660) [ 1624.257822] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1625.080456] Lustre: Failing over lustre-MDT0000 [ 1625.265066] Lustre: server umount lustre-MDT0000 complete [ 1640.150109] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1640.152041] LDISKFS-fs (dm-0): recovery complete [ 1640.156320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.829984] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1645.567228] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 1645.567327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 1646.970514] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1647.685506] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1651.693487] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 01:34:51 (1775885691) [ 1653.642635] Lustre: 71814:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e08df21d-d694-4084-80c8-4b7c9f2345cf at adminstrative request [ 1657.383166] Lustre: Failing over lustre-MDT0000 [ 1657.634129] Lustre: server umount lustre-MDT0000 complete [ 1670.934221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.620486] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1676.260850] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1676.264204] Lustre: Skipped 62 previous similar messages [ 1676.276373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1676.277024] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 1677.586147] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1678.223944] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1680.640307] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1684.091431] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1686.503446] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 01:35:26 (1775885726) [ 1686.924386] Lustre: 73713:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e08df21d-d694-4084-80c8-4b7c9f2345cf at adminstrative request [ 1692.263859] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 01:35:31 (1775885731) [ 1695.161910] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1695.928052] Lustre: Failing over lustre-MDT0000 [ 1696.041471] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 1696.125025] Lustre: server umount lustre-MDT0000 complete [ 1710.490817] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1710.492697] LDISKFS-fs (dm-0): recovery complete [ 1710.496600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1711.915353] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1715.720633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 1715.720898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 1716.969357] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1717.574833] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1721.294288] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 01:36:00 (1775885760) [ 1723.986572] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1724.658810] Lustre: Failing over lustre-MDT0000 [ 1724.861693] Lustre: server umount lustre-MDT0000 complete [ 1739.459945] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1739.461970] LDISKFS-fs (dm-0): recovery complete [ 1739.466035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.054734] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1744.891549] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1744.891587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1746.143834] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1746.778327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1750.460381] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 01:36:30 (1775885790) [ 1753.077222] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1753.778822] Lustre: Failing over lustre-MDT0000 [ 1754.007951] Lustre: server umount lustre-MDT0000 complete [ 1768.740041] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1768.741249] LDISKFS-fs (dm-0): recovery complete [ 1768.745951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.335072] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1774.099061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1774.099545] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1775.633619] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1776.381838] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1780.465986] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 01:37:00 (1775885820) [ 1783.320515] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1784.035609] Lustre: Failing over lustre-MDT0000 [ 1784.210478] Lustre: server umount lustre-MDT0000 complete [ 1784.289361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1784.294530] Lustre: Skipped 62 previous similar messages [ 1799.643603] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1799.645493] LDISKFS-fs (dm-0): recovery complete [ 1799.651504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.590389] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1803.559685] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1803.566111] Lustre: Skipped 15 previous similar messages [ 1805.302647] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1805.308709] Lustre: Skipped 15 previous similar messages [ 1805.329266] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1805.329465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1806.817721] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1807.532415] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1811.800708] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 01:37:31 (1775885851) [ 1815.237555] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1816.059596] Lustre: Failing over lustre-MDT0000 [ 1816.305711] Lustre: server umount lustre-MDT0000 complete [ 1820.642555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1820.646460] LustreError: Skipped 14 previous similar messages [ 1831.540043] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1831.543038] LDISKFS-fs (dm-0): recovery complete [ 1831.549760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.615603] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1831.622281] LustreError: Skipped 15 previous similar messages [ 1833.396267] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1837.059661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 1837.060385] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 1838.776105] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1839.574744] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1844.020885] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 01:38:03 (1775885883) [ 1847.277083] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1848.221392] Lustre: Failing over lustre-MDT0000 [ 1848.401617] Lustre: server umount lustre-MDT0000 complete [ 1863.472645] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1863.474666] LDISKFS-fs (dm-0): recovery complete [ 1863.478229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.201905] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1868.888400] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1868.889551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1870.590189] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1871.299747] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1875.317606] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 01:38:34 (1775885914) [ 1878.342412] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1879.150853] Lustre: Failing over lustre-MDT0000 [ 1879.261531] Lustre: server umount lustre-MDT0000 complete [ 1894.423076] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1894.424789] LDISKFS-fs (dm-0): recovery complete [ 1894.430048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.168870] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1900.055477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1900.056703] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1901.571678] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1902.274614] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1906.137785] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 01:39:05 (1775885945) [ 1909.096621] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1909.919130] Lustre: Failing over lustre-MDT0000 [ 1910.123362] Lustre: server umount lustre-MDT0000 complete [ 1924.656378] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1924.658173] LDISKFS-fs (dm-0): recovery complete [ 1924.662315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1924.813801] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1924.818675] Lustre: Skipped 15 previous similar messages [ 1926.354762] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1930.243411] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1930.243504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1931.697937] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1932.416485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1936.516454] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 01:39:36 (1775885976) [ 1939.434217] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1940.339877] Lustre: Failing over lustre-MDT0000 [ 1940.448595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.452618] Lustre: Skipped 3 previous similar messages [ 1940.496771] Lustre: server umount lustre-MDT0000 complete [ 1955.325784] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1955.327372] LDISKFS-fs (dm-0): recovery complete [ 1955.335653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.911187] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1960.959818] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1960.959828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1962.342860] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1963.083978] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1967.199613] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 01:40:06 (1775886006) [ 1970.556517] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1971.324283] Lustre: Failing over lustre-MDT0000 [ 1971.487594] Lustre: server umount lustre-MDT0000 complete [ 1986.722813] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1986.724466] LDISKFS-fs (dm-0): recovery complete [ 1986.731641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.411701] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1992.190570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1992.190612] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1993.758864] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1994.617677] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1999.079937] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 01:40:38 (1775886038) [ 2002.339446] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2003.167749] Lustre: Failing over lustre-MDT0000 [ 2003.340138] Lustre: server umount lustre-MDT0000 complete [ 2018.261264] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2018.263085] LDISKFS-fs (dm-0): recovery complete [ 2018.266781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2019.916828] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2023.422262] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 2023.422509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 2025.019149] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2025.862485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2030.858126] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 01:41:10 (1775886070) [ 2031.299274] Lustre: 94066:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting e08df21d-d694-4084-80c8-4b7c9f2345cf at adminstrative request [ 2036.750895] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 01:41:16 (1775886076) [ 2039.423836] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2040.169514] Lustre: Failing over lustre-MDT0000 [ 2040.279136] Lustre: server umount lustre-MDT0000 complete [ 2045.515095] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2045.518897] LDISKFS-fs (dm-0): recovery complete [ 2045.523691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.690269] Lustre: lustre-MDT0000: Aborting client recovery [ 2045.692866] LustreError: 95346:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2045.696130] Lustre: 95383:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2045.700237] Lustre: 95383:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client e08df21d-d694-4084-80c8-4b7c9f2345cf@ [ 2045.705233] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2045.709830] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2045.717917] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2045.739995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 2045.740201] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 2047.318385] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2051.045377] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2059.586261] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 01:41:39 (1775886099) [ 2062.368206] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2063.131692] Lustre: Failing over lustre-MDT0000 [ 2063.309634] Lustre: server umount lustre-MDT0000 complete [ 2068.450624] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2068.453451] LDISKFS-fs (dm-0): recovery complete [ 2068.457995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.610082] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2068.646874] Lustre: lustre-MDT0000: Aborting client recovery [ 2068.648856] LustreError: 97198:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2068.651801] Lustre: 97235:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2068.656160] Lustre: 97235:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2068.659892] Lustre: 97235:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2068.664666] Lustre: 97235:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2068.668219] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2068.676554] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 2068.686248] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 2068.717048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1537) [ 2068.717346] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1537) [ 2070.317983] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2074.081692] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2080.025113] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2080.027068] Lustre: Skipped 1 previous similar message [ 2082.553683] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 01:42:02 (1775886122) [ 2085.335857] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2085.994960] Lustre: Failing over lustre-MDT0000 [ 2086.086442] Lustre: server umount lustre-MDT0000 complete [ 2090.598857] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2090.600313] LDISKFS-fs (dm-0): recovery complete [ 2090.603023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.719852] Lustre: lustre-MDT0000: Aborting client recovery [ 2090.721434] LustreError: 99053:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2090.722944] Lustre: 99090:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2090.727893] Lustre: 99090:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2090.730589] Lustre: 99090:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2090.733658] Lustre: 99090:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2090.736390] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2090.739774] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 2090.746257] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 2090.766840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 2090.766844] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 2092.003621] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2096.098665] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2104.920501] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 01:42:24 (1775886144) [ 2105.237796] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2105.239184] LustreError: 8410:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b7767f9c00 x1862149478576000/t201863462916(0) o36->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:72/0 lens 512/456 e 0 to 0 dl 1775886157 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 2107.843762] Lustre: Failing over lustre-MDT0000 [ 2107.933138] Lustre: server umount lustre-MDT0000 complete [ 2110.250484] LustreError: 8842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.255602] LustreError: 8842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 252 previous similar messages [ 2110.804843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.935560] Lustre: lustre-MDT0000: Aborting client recovery [ 2110.936928] LustreError: 100367:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2110.939298] Lustre: 100403:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2110.942813] Lustre: 100403:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2110.945515] Lustre: 100403:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2110.949108] Lustre: 100403:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2110.951825] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2110.954699] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 2110.960491] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 2110.978171] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 2110.978178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1601) [ 2112.215503] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2116.066705] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2125.410308] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 2125.974054] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 01:42:45 (1775886165) [ 2128.504718] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2129.424914] Lustre: Failing over lustre-MDT0000 [ 2129.608081] Lustre: server umount lustre-MDT0000 complete [ 2133.527278] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2133.528525] LDISKFS-fs (dm-0): recovery complete [ 2133.531993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.654888] Lustre: lustre-MDT0000: Aborting client recovery [ 2133.656780] LustreError: 102317:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2133.659627] Lustre: 102354:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2133.665024] Lustre: 102354:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2133.668177] Lustre: 102354:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2133.671518] Lustre: 102354:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2133.673752] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2133.677042] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 2133.683358] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 2133.703877] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 2133.703986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1633) [ 2135.105748] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2139.104912] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2147.225358] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 01:43:06 (1775886186) [ 2158.702690] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2159.400282] Lustre: Failing over lustre-MDT0000 [ 2159.583935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2159.586300] Lustre: Skipped 3 previous similar messages [ 2159.639464] Lustre: server umount lustre-MDT0000 complete [ 2173.970929] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2173.972935] LDISKFS-fs (dm-0): recovery complete [ 2173.976270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.085897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.088046] Lustre: Skipped 20 previous similar messages [ 2175.379325] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2179.598895] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 2179.599033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 2180.816043] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2181.398831] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2190.009868] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 01:43:49 (1775886229) [ 2197.806933] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2200.657935] Lustre: Failing over lustre-MDT0000 [ 2200.944150] Lustre: server umount lustre-MDT0000 complete [ 2215.299794] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2215.301599] LDISKFS-fs (dm-0): recovery complete [ 2215.304795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.686203] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2221.032438] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 2221.032905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 2222.154155] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2222.681545] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2230.946180] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 01:44:30 (1775886270) [ 2231.738328] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2232.091337] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2232.094541] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2234.364669] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 01:44:33 (1775886273) [ 2240.707563] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2243.755502] Lustre: Failing over lustre-OST0000 [ 2243.806948] Lustre: server umount lustre-OST0000 complete [ 2258.132545] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 2258.133718] LDISKFS-fs (dm-2): recovery complete [ 2258.136358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2259.983593] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2305.899575] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 01:45:45 (1775886345) [ 2308.752546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2309.706223] Lustre: Failing over lustre-MDT0000 [ 2309.894282] Lustre: server umount lustre-MDT0000 complete [ 2324.079110] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2324.081146] LDISKFS-fs (dm-0): recovery complete [ 2324.084633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2325.490457] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2329.572422] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2329.575206] Lustre: Skipped 75 previous similar messages [ 2329.593609] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2329.593695] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 2330.705442] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2331.271487] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2344.940179] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 01:46:24 (1775886384) [ 2345.951107] Lustre: 111433:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775886370/real 1775886370] req@ffff94b74fe15180 x1862149494216192/t0(0) o5->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1775886386 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-0-0.0' uid:0 gid:0 projid:4294967295 [ 2345.962210] LustreError: 111433:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2345.963199] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2346.688139] LustreError: 8410:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2346.975663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 2352.095163] LustreError: 8410:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2352.099242] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnecting [ 2352.709230] LustreError: 20722:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2357.727208] LustreError: 20722:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2357.730505] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnecting [ 2358.361134] LustreError: 8410:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2363.871127] LustreError: 8410:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2363.875094] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnecting [ 2364.482456] LustreError: 20722:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2369.503134] LustreError: 20722:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2370.065647] LustreError: 6509:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2375.135082] LustreError: 6509:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2375.137470] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnecting [ 2375.140244] Lustre: Skipped 1 previous similar message [ 2381.360056] LustreError: 8842:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2381.362215] LustreError: 8842:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 2386.399124] LustreError: 8842:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2386.401767] LustreError: 8842:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 2392.543162] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnecting [ 2392.547665] Lustre: Skipped 2 previous similar messages [ 2399.294778] LustreError: 6509:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2399.297023] LustreError: 6509:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 2404.319120] LustreError: 6509:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2404.321346] LustreError: 6509:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 2407.267889] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 01:47:26 (1775886446) [ 2407.924593] LustreError: 6509:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2418.470202] Lustre: lustre-MDT0000: Export ffff94b74fb4a000 already connecting from 192.168.204.30@tcp [ 2423.590091] Lustre: lustre-MDT0000: Export ffff94b74fb4a000 already connecting from 192.168.204.30@tcp [ 2428.710219] Lustre: lustre-MDT0000: Export ffff94b74fb4a000 already connecting from 192.168.204.30@tcp [ 2433.829969] Lustre: lustre-MDT0000: Export ffff94b74fb4a000 already connecting from 192.168.204.30@tcp [ 2433.832991] Lustre: Skipped 1 previous similar message [ 2438.950032] Lustre: lustre-MDT0000: Export ffff94b74fb4a000 already connecting from 192.168.204.30@tcp [ 2447.967079] LustreError: 6509:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2447.970534] Lustre: 6509:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94b74441ca80 x1862149481206912/t0(0) o38->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:0/0 lens 520/416 e 0 to 0 dl 1775886468 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2449.189971] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnecting [ 2449.193461] Lustre: Skipped 3 previous similar messages [ 2449.195200] LustreError: 20722:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2474.790047] Lustre: lustre-MDT0000: Export ffff94b74fb4a000 already connecting from 192.168.204.30@tcp [ 2474.792524] Lustre: Skipped 1 previous similar message [ 2489.239065] LustreError: 20722:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2489.241757] Lustre: 20722:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94b74441e300 x1862149481212160/t0(0) o38->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:0/0 lens 520/416 e 0 to 0 dl 1775886510 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2490.150119] LustreError: 8410:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2515.749843] Lustre: lustre-MDT0000: Export ffff94b74fb4a000 already connecting from 192.168.204.30@tcp [ 2515.751864] Lustre: Skipped 2 previous similar messages [ 2530.191105] LustreError: 8410:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2530.193757] Lustre: 8410:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94b74341ce00 x1862149481215232/t0(0) o38->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:0/0 lens 520/416 e 0 to 0 dl 1775886551 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2531.110091] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnecting [ 2531.112871] Lustre: Skipped 1 previous similar message [ 2531.114162] LustreError: 20722:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2556.709857] Lustre: lustre-MDT0000: Export ffff94b74fb4a000 already connecting from 192.168.204.30@tcp [ 2556.711984] Lustre: Skipped 2 previous similar messages [ 2571.159102] LustreError: 20722:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2571.161896] Lustre: 20722:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94b7447a3b80 x1862149481218304/t0(0) o38->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:0/0 lens 520/416 e 0 to 0 dl 1775886592 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2572.070058] LustreError: 8410:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2612.111126] LustreError: 8410:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2612.114412] Lustre: 8410:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94b776c07800 x1862149481221376/t0(0) o38->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:0/0 lens 520/416 e 0 to 0 dl 1775886633 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2613.029869] LustreError: 6508:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2618.023121] LustreError: 6508:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout interrupted [ 2619.909132] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 01:50:59 (1775886659) [ 2623.075949] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2624.901096] Lustre: Failing over lustre-MDT0000 [ 2625.011430] Lustre: server umount lustre-MDT0000 complete [ 2626.527424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2626.530210] LustreError: Skipped 15 previous similar messages [ 2626.532457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.537065] Lustre: Skipped 65 previous similar messages [ 2629.417331] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2629.419243] LDISKFS-fs (dm-0): recovery complete [ 2629.422457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.458681] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2629.461312] LustreError: Skipped 14 previous similar messages [ 2629.517846] Lustre: *** cfs_fail_loc=712, val=0*** [ 2629.520036] LustreError: 42166:0:(service.c:1390:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff94b7444a2a00 x1862149494359936/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2629.577131] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2629.577215] Lustre: lustre-MDT0000: Aborting client recovery [ 2629.579921] Lustre: Skipped 22 previous similar messages [ 2629.582079] LustreError: 116167:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2629.583103] Lustre: 116203:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2629.583107] Lustre: 116203:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2629.592158] Lustre: 116203:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2629.596113] Lustre: 116203:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2629.599143] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2629.602548] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2629.609193] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2629.628277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2629.628311] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2630.942424] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2634.720169] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2641.438662] Lustre: Failing over lustre-MDT0000 [ 2641.636871] Lustre: server umount lustre-MDT0000 complete [ 2654.309681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2655.724295] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2659.117558] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2659.121837] Lustre: Skipped 11 previous similar messages [ 2659.819496] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2659.823216] Lustre: Skipped 11 previous similar messages [ 2659.837381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2659.837458] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2660.991652] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2661.534945] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2664.736685] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 01:51:44 (1775886704) [ 2664.777833] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnecting [ 2664.781021] Lustre: Skipped 2 previous similar messages [ 2667.709737] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 01:51:47 (1775886707) [ 2668.015160] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2668.016958] LustreError: 6517:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b77c2bdf80 x1862149481277568/t0(0) o700->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:634/0 lens 264/248 e 0 to 0 dl 1775886719 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2685.330143] Lustre: Failing over lustre-MDT0000 [ 2685.408652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2685.412847] Lustre: Skipped 3 previous similar messages [ 2685.550852] Lustre: server umount lustre-MDT0000 complete [ 2698.471347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2700.009264] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2703.865959] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2703.866040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2705.083335] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2705.655213] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2709.658940] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 01:52:29 (1775886749) [ 2710.474830] Lustre: Failing over lustre-OST0000 [ 2710.550890] Lustre: server umount lustre-OST0000 complete [ 2710.823549] LustreError: 8852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.830216] LustreError: 8852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 77 previous similar messages [ 2723.303082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2725.171920] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2728.231424] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2728.757230] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2793.743065] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 01:53:53 (1775886833) [ 2796.123281] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2796.751916] Lustre: Failing over lustre-MDT0000 [ 2796.967379] Lustre: server umount lustre-MDT0000 complete [ 2810.710328] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2810.711521] LDISKFS-fs (dm-0): recovery complete [ 2810.713954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2810.811974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2810.814464] Lustre: Skipped 7 previous similar messages [ 2812.006806] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2816.053310] Lustre: *** cfs_fail_loc=216, val=0*** [ 2816.053330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2816.055167] LustreError: 121992:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2817.119397] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2878.548962] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 01:55:18 (1775886918) [ 2886.534361] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 01:55:26 (1775886926) [ 2887.499536] Lustre: Failing over lustre-MDT0000 [ 2887.648593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.652259] Lustre: Skipped 4 previous similar messages [ 2887.693795] Lustre: server umount lustre-MDT0000 complete [ 2900.458641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2901.806433] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2905.571777] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2905.573145] LustreError: 123564:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b77c3d0a80 x1862149481395328/t0(0) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:161/0 lens 328/344 e 0 to 0 dl 1775887001 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2965.798025] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 1:02 [ 2965.803661] Lustre: 123564:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2965.810319] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2965.812788] Lustre: Skipped 27 previous similar messages [ 2965.821788] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 2965.821789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 2967.025863] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2967.515418] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2971.063064] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 01:56:50 (1775887010) [ 2972.416795] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2974.942789] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2975.591485] Lustre: Failing over lustre-MDT0000 [ 2975.751566] Lustre: server umount lustre-MDT0000 complete [ 2989.495050] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2989.496168] LDISKFS-fs (dm-0): recovery complete [ 2989.498417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.787383] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2994.673183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3137) [ 2994.673234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3105) [ 2995.839889] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2996.378832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2999.550732] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 01:57:19 (1775887039) [ 2999.890816] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3003.327829] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3003.915203] Lustre: Failing over lustre-MDT0000 [ 3003.999438] Lustre: server umount lustre-MDT0000 complete [ 3017.949494] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3017.951196] LDISKFS-fs (dm-0): recovery complete [ 3017.954859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.022170] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (not set up) [ 3019.265537] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3023.347345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 3023.347359] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3137) [ 3024.454899] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3024.957838] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3028.151744] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 01:57:47 (1775887067) [ 3028.480978] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3032.070499] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3032.675553] Lustre: Failing over lustre-MDT0000 [ 3032.768621] Lustre: server umount lustre-MDT0000 complete [ 3046.759383] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3046.761051] LDISKFS-fs (dm-0): recovery complete [ 3046.763867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.187105] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3052.020027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3201) [ 3052.020032] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 3056.429623] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 01:58:16 (1775887096) [ 3057.797529] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3057.799388] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 3057.800799] LustreError: 6510:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b74fe56300 x1862149481446144/t257698037777(0) o35->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:269/0 lens 392/456 e 0 to 0 dl 1775887109 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3058.722278] Lustre: Failing over lustre-MDT0000 [ 3058.971283] Lustre: server umount lustre-MDT0000 complete [ 3071.628537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3072.944253] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3077.103421] Lustre: 6511:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b744574700 x1862149481446144/t257698037777(0) o35->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:289/0 lens 392/456 e 0 to 0 dl 1775887129 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3077.108402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3233) [ 3077.108408] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 3078.276990] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3078.833564] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3082.286560] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 01:58:41 (1775887121) [ 3082.636140] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3082.637712] LustreError: 20722:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b748767480 x1862149481458944/t261993005072(0) o36->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:338/0 lens 504/448 e 0 to 0 dl 1775887178 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3086.259528] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3086.898248] Lustre: Failing over lustre-MDT0000 [ 3086.983445] Lustre: server umount lustre-MDT0000 complete [ 3101.117100] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3101.118312] LDISKFS-fs (dm-0): recovery complete [ 3101.121041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.589666] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3106.285439] Lustre: 8410:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b7444a4380 x1862149481458944/t261993005072(0) o36->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:362/0 lens 504/2880 e 0 to 0 dl 1775887202 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3106.293032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 3106.293041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233) [ 3107.464277] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3108.008045] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3111.623529] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 01:59:11 (1775887151) [ 3111.973326] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3111.974879] LustreError: 20722:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b7444a4000 x1862149481471616/t266287972368(0) o36->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:367/0 lens 504/448 e 0 to 0 dl 1775887207 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3113.251426] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3115.540227] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3116.109772] Lustre: Failing over lustre-MDT0000 [ 3116.186830] Lustre: server umount lustre-MDT0000 complete [ 3129.972672] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3129.974503] LDISKFS-fs (dm-0): recovery complete [ 3129.978119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3131.303892] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3135.469189] Lustre: 6511:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b74fc52680 x1862149481472384/t266287972369(0) o35->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:347/0 lens 392/456 e 0 to 0 dl 1775887187 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3135.474680] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 3135.475129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297) [ 3135.477080] Lustre: 6511:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 3139.693059] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 01:59:39 (1775887179) [ 3140.026747] LustreError: 6507:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b77c3da300 x1862149481484288/t270582939664(0) o36->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:395/0 lens 504/448 e 0 to 0 dl 1775887235 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3140.032602] LustreError: 6507:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 3141.291612] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3141.294256] Lustre: Skipped 1 previous similar message [ 3143.814680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3144.457942] Lustre: Failing over lustre-MDT0000 [ 3144.539735] Lustre: server umount lustre-MDT0000 complete [ 3158.712034] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3158.713778] LDISKFS-fs (dm-0): recovery complete [ 3158.717536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.160316] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3164.143628] Lustre: 6509:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b642042a00 x1862149481484288/t270582939664(0) o36->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:420/0 lens 504/2880 e 0 to 0 dl 1775887260 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3164.149240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3329) [ 3164.149257] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 3168.160271] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 02:00:07 (1775887207) [ 3168.506859] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3169.788462] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3169.790483] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 3169.792377] Lustre: Skipped 2 previous similar messages [ 3173.238554] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3173.881633] Lustre: Failing over lustre-MDT0000 [ 3173.967308] Lustre: server umount lustre-MDT0000 complete [ 3188.072762] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3188.074241] LDISKFS-fs (dm-0): recovery complete [ 3188.077540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3189.522100] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3193.327297] Lustre: 6511:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b74458ed80 x1862149481496704/t274877906960(0) o35->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:405/0 lens 392/456 e 0 to 0 dl 1775887245 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3193.334706] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 3193.334726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 3197.804523] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 02:00:37 (1775887237) [ 3198.149319] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 3198.151498] LustreError: 6509:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b648d2c380 x1862149481506560/t279172874255(0) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:454/0 lens 664/608 e 0 to 0 dl 1775887294 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 3198.158919] LustreError: 6509:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 3258.669292] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnecting [ 3258.673218] Lustre: Skipped 4 previous similar messages [ 3258.675698] Lustre: 20722:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b7486f5f80 x1862149481506560/t279172874255(0) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:514/0 lens 664/3488 e 0 to 0 dl 1775887354 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 3260.835950] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 02:01:40 (1775887300) [ 3263.404504] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3264.059078] Lustre: Failing over lustre-MDT0000 [ 3264.143828] Lustre: server umount lustre-MDT0000 complete [ 3264.991696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.996107] Lustre: Skipped 58 previous similar messages [ 3264.997531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3264.999888] LustreError: Skipped 13 previous similar messages [ 3278.257918] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3278.259219] LDISKFS-fs (dm-0): recovery complete [ 3278.262996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3278.297191] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3278.301378] LustreError: Skipped 12 previous similar messages [ 3278.403994] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3278.406482] Lustre: Skipped 15 previous similar messages [ 3279.142081] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3279.146139] Lustre: Skipped 12 previous similar messages [ 3279.681957] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3283.436099] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3283.438873] Lustre: Skipped 12 previous similar messages [ 3283.453608] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 3283.453621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3393) [ 3284.629825] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3285.178192] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3298.725457] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 02:02:18 (1775887338) [ 3301.495757] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3302.109621] Lustre: Failing over lustre-MDT0000 [ 3302.275200] Lustre: server umount lustre-MDT0000 complete [ 3314.144929] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.151202] LustreError: 6512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 173 previous similar messages [ 3316.348984] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3316.351102] LDISKFS-fs (dm-0): recovery complete [ 3316.354051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.773546] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3321.850723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 3321.850733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3425) [ 3323.045297] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3323.606950] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3325.958946] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3331.150826] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 02:02:50 (1775887370) [ 3341.283436] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3341.923136] Lustre: Failing over lustre-MDT0000 [ 3342.246904] Lustre: server umount lustre-MDT0000 complete [ 3356.390981] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3356.392535] LDISKFS-fs (dm-0): recovery complete [ 3356.395966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.835158] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3361.872302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 3361.872302] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 3362.977186] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3363.476881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3376.619404] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 02:03:36 (1775887416) [ 3380.219225] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3380.801096] Lustre: Failing over lustre-MDT0000 [ 3380.984602] Lustre: server umount lustre-MDT0000 complete [ 3394.909433] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3394.910809] LDISKFS-fs (dm-0): recovery complete [ 3394.913642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.255281] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3400.186816] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 3400.186861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 3401.247373] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3401.754519] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3404.503765] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3404.987804] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3406.963926] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 02:04:06 (1775887446) [ 3412.770080] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3473.440414] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3473.441789] Lustre: Skipped 1 previous similar message [ 3473.442943] LustreError: 8410:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b747d84050 x1862149484292608/t296352743435(0) o36->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:729/0 lens 66040/440 e 0 to 0 dl 1775887569 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3533.096049] Lustre: 8410:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b74fc51f80 x1862149484292608/t296352743435(0) o36->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:34/0 lens 66040/440 e 0 to 0 dl 1775887629 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3536.035557] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3536.591574] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 02:06:16 (1775887576) [ 3540.734212] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3541.506228] Lustre: Failing over lustre-MDT0000 [ 3541.703064] Lustre: server umount lustre-MDT0000 complete [ 3555.568669] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3555.569881] LDISKFS-fs (dm-0): recovery complete [ 3555.572674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.674158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.676334] Lustre: Skipped 13 previous similar messages [ 3556.958085] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3561.093668] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 3561.093674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 3562.173594] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3562.688642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3566.397274] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 02:06:46 (1775887606) [ 3571.820572] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3574.802101] Lustre: Failing over lustre-OST0000 [ 3574.839240] Lustre: server umount lustre-OST0000 complete [ 3588.857088] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3588.859278] LDISKFS-fs (dm-2): recovery complete [ 3588.862751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3590.192067] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3590.195529] Lustre: Skipped 52 previous similar messages [ 3590.855041] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3602.288624] Lustre: Failing over lustre-OST0000 [ 3602.333990] Lustre: server umount lustre-OST0000 complete [ 3615.025377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3616.964615] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3620.086682] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3620.621919] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3654.549856] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 02:08:14 (1775887694) [ 3655.435996] Lustre: Failing over lustre-MDT0000 [ 3655.742233] Lustre: server umount lustre-MDT0000 complete [ 3668.443198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.857810] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3673.587513] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 3673.587512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 3681.342263] Lustre: Failing over lustre-MDT0000 [ 3681.575395] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.30@tcp (stopping) [ 3681.611530] Lustre: server umount lustre-MDT0000 complete [ 3694.451329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3695.885741] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3699.698852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 3699.698961] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 3700.874533] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3701.425434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3704.820167] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 02:09:04 (1775887744) [ 3715.977978] Lustre: Failing over lustre-OST0000 [ 3716.030353] Lustre: server umount lustre-OST0000 complete [ 3728.723386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3730.597682] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3733.665478] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3734.228419] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3738.212670] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 02:09:37 (1775887777) [ 3738.868352] Lustre: Failing over lustre-MDT0000 [ 3739.216959] Lustre: server umount lustre-MDT0000 complete [ 3742.415895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.449879] Lustre: *** cfs_fail_loc=605, val=0*** [ 3742.451031] LustreError: 157352:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc078da30 failed: rc = -95 [ 3742.453531] LustreError: 157352:0:(obd_config.c:843:class_setup()) setup MGS failed (-95) [ 3742.455149] LustreError: 157352:0:(obd_mount.c:250:lustre_start_simple()) MGS setup error -95 [ 3742.457572] LustreError: 157352:0:(tgt_mount.c:114:server_deregister_mount()) MGS not registered [ 3742.459710] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3742.462400] LustreError: 157352:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 3742.467485] Lustre: server umount lustre-MDT0000 complete [ 3742.469469] LustreError: 157352:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3745.029973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3746.473718] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3749.201179] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 02:09:48 (1775887788) [ 3750.395506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3750.395516] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3752.913309] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3753.945994] Lustre: Failing over lustre-MDT0000 [ 3754.255766] Lustre: server umount lustre-MDT0000 complete [ 3768.287774] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3768.289938] LDISKFS-fs (dm-0): recovery complete [ 3768.294298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3768.615441] Lustre: *** cfs_fail_loc=707, val=0*** [ 3769.781192] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3828.014954] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 3828.029831] Lustre: 159514:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3828.033617] Lustre: 159514:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 3828.130749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 3828.130767] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 3829.413811] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3829.945424] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3833.627633] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 02:11:13 (1775887873) [ 3857.181753] LustreError: 20722:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b74458f800 x1862149485171072/t0(0) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:314/0 lens 664/0 e 0 to 0 dl 1775887909 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3857.190972] LustreError: 20722:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3868.215121] LustreError: 20722:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3868.221238] LustreError: 6510:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b776acd880 x1862149485172480/t0(0) o35->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:325/0 lens 392/0 e 0 to 0 dl 1775887920 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3873.760981] LustreError: 42166:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b7768c0380 x1862149496024064/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:330/0 lens 544/0 e 0 to 0 dl 1775887925 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3873.768134] LustreError: 42166:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 48 previous similar messages [ 3879.974478] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 02:11:59 (1775887919) [ 3903.231529] LustreError: 41102:0:(tgt_handler.c:2810:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3914.255072] LustreError: 41102:0:(tgt_handler.c:2810:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3917.239912] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 02:12:36 (1775887956) [ 3940.225487] LustreError: 6509:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b744304000 x1862149485195264/t0(0) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:397/0 lens 576/0 e 0 to 0 dl 1775887992 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3940.233517] LustreError: 6509:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3945.335047] LustreError: 6509:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3945.339755] LustreError: 6507:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b7493fdc00 x1862149485196032/t0(0) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:402/0 lens 664/0 e 0 to 0 dl 1775887997 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3956.144101] LustreError: 20722:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3956.149606] LustreError: 6508:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b644ee9500 x1862149485217536/t0(0) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:453/0 lens 664/0 e 0 to 0 dl 1775888048 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3956.161343] LustreError: 6508:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 3968.044304] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 02:13:27 (1775888007) [ 4062.831854] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 02:15:02 (1775888102) [ 4085.630557] LustreError: 8410:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b7444a7b80 x1862149485273344/t0(0) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:542/0 lens 576/0 e 0 to 0 dl 1775888137 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4085.639958] LustreError: 8410:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 4085.643304] LustreError: 8410:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 4085.647281] LustreError: 8410:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 4086.063134] LustreError: 8410:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4101.881105] LustreError: 6508:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 4101.885773] LustreError: 6508:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 4102.303099] LustreError: 6508:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4102.305825] LustreError: 6508:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 4117.713103] LustreError: 20722:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b74458d500 x1862149485293312/t0(0) o36->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:574/0 lens 504/0 e 0 to 0 dl 1775888169 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4117.722320] LustreError: 20722:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 81 previous similar messages [ 4129.805360] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 02:16:09 (1775888169) [ 4154.320836] Lustre: DEBUG MARKER: phase 2 [ 4157.531731] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 02:16:37 (1775888197) [ 4227.549557] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 02:17:47 (1775888267) [ 4228.026507] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 4228.546874] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 02:17:48 (1775888268) [ 4230.319746] Lustre: DEBUG MARKER: Started rundbench load pid=134196 ... [ 4234.174262] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4235.681194] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 4236.352404] Lustre: Failing over lustre-MDT0000 [ 4236.648329] Lustre: server umount lustre-MDT0000 complete [ 4237.307456] LustreError: 20722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4237.315160] LustreError: 20722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 123 previous similar messages [ 4237.792442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4237.797412] LustreError: Skipped 9 previous similar messages [ 4237.800354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.809043] Lustre: Skipped 38 previous similar messages [ 4251.135980] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4251.137810] LDISKFS-fs (dm-0): recovery complete [ 4251.142805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.191696] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4251.195649] LustreError: Skipped 8 previous similar messages [ 4251.278300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4251.280757] Lustre: Skipped 7 previous similar messages [ 4251.295400] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4251.297712] Lustre: Skipped 11 previous similar messages [ 4252.711860] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4255.013631] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4255.016776] Lustre: Skipped 11 previous similar messages [ 4256.741555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4256.743642] Lustre: Skipped 21 previous similar messages [ 4256.750137] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4256.754083] Lustre: Skipped 11 previous similar messages [ 4256.770775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 4256.771045] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 4257.970121] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4258.550918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4263.361620] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4264.935455] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 4265.617955] Lustre: Failing over lustre-MDT0001 [ 4265.627440] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.30@tcp (stopping) [ 4265.630750] Lustre: Skipped 4 previous similar messages [ 4265.888486] Lustre: server umount lustre-MDT0001 complete [ 4280.063195] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 4280.064521] LDISKFS-fs (dm-1): recovery complete [ 4280.069458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4281.434538] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4285.766028] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:819 to 0x280000400:865) [ 4285.766028] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:801 to 0x2c0000400:833) [ 4287.049254] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4287.696585] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4292.573614] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4294.109829] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 4294.781212] Lustre: Failing over lustre-MDT0000 [ 4295.028397] Lustre: server umount lustre-MDT0000 complete [ 4309.290608] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4309.293210] LDISKFS-fs (dm-0): recovery complete [ 4309.297224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4310.837969] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4314.617406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 4314.617643] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 4315.836611] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4316.388975] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4321.279340] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4322.836930] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 4323.518997] Lustre: Failing over lustre-MDT0001 [ 4323.702454] Lustre: server umount lustre-MDT0001 complete [ 4337.767037] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 4337.768394] LDISKFS-fs (dm-1): recovery complete [ 4337.772740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4339.177607] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4343.485381] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1223 to 0x2c0000400:1249) [ 4343.485389] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1256 to 0x280000400:1281) [ 4344.761990] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4345.362648] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4350.354608] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4351.910140] Lustre: DEBUG MARKER: test_70b fail mds1 5 times [ 4352.564056] Lustre: Failing over lustre-MDT0000 [ 4352.862774] Lustre: server umount lustre-MDT0000 complete [ 4367.216823] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4367.219028] LDISKFS-fs (dm-0): recovery complete [ 4367.222701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.853793] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4372.481909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5153) [ 4372.481910] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5121) [ 4373.860246] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4374.441401] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4377.877566] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 02:20:17 (1775888417) [ 4500.901165] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4511.696886] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 4512.384141] Lustre: Failing over lustre-MDT0000 [ 4517.987033] Lustre: server umount lustre-MDT0000 complete [ 4532.211633] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4532.213766] LDISKFS-fs (dm-0): recovery complete [ 4532.217559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.646466] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4539.804282] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6478 to 0x2c0000401:6497) [ 4539.810222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6510 to 0x280000401:6529) [ 4541.065890] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4541.742808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4665.988556] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4676.733286] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 4677.339930] Lustre: Failing over lustre-MDT0000 [ 4682.834123] Lustre: server umount lustre-MDT0000 complete [ 4696.566196] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4696.567383] LDISKFS-fs (dm-0): recovery complete [ 4696.570019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.878546] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4704.398019] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7977 to 0x2c0000401:8001) [ 4704.398048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8009 to 0x280000401:8033) [ 4705.559183] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4706.173768] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4728.422974] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 02:26:08 (1775888768) [ 4851.386043] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4862.146159] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 4862.821878] Lustre: Failing over lustre-MDT0000 [ 4863.455628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4863.458443] LustreError: Skipped 6 previous similar messages [ 4863.460053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.463798] Lustre: Skipped 25 previous similar messages [ 4866.016187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4866.019062] Lustre: Skipped 17 previous similar messages [ 4869.197869] Lustre: server umount lustre-MDT0000 complete [ 4871.135554] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4871.141891] LustreError: 6509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 103 previous similar messages [ 4883.339213] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4883.340980] LDISKFS-fs (dm-0): recovery complete [ 4883.344870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.384745] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4883.388692] LustreError: Skipped 4 previous similar messages [ 4883.467513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4883.469820] Lustre: Skipped 6 previous similar messages [ 4883.482023] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4883.484380] Lustre: Skipped 6 previous similar messages [ 4884.758181] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4887.846107] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4887.849618] Lustre: Skipped 6 previous similar messages [ 4888.545636] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4888.547393] Lustre: Skipped 25 previous similar messages [ 4889.664257] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4889.668163] Lustre: Skipped 6 previous similar messages [ 4889.684343] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8605 to 0x2c0000401:8641) [ 4889.698232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8637 to 0x280000401:8673) [ 4890.956685] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4891.528506] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5015.508546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5026.233077] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 5026.862151] Lustre: Failing over lustre-MDT0000 [ 5033.029617] Lustre: server umount lustre-MDT0000 complete [ 5046.973960] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5046.975440] LDISKFS-fs (dm-0): recovery complete [ 5046.978823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5048.325526] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5053.480061] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9090 to 0x2c0000401:9121) [ 5053.480233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9121 to 0x280000401:9153) [ 5054.737350] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5055.295457] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5058.586792] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 02:31:38 (1775889098) [ 5181.619037] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5192.334404] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 5192.958736] Lustre: Failing over lustre-MDT0000 [ 5193.243642] Lustre: server umount lustre-MDT0000 complete [ 5207.151457] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5207.152707] LDISKFS-fs (dm-0): recovery complete [ 5207.155460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5208.552433] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5215.964950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11254 to 0x280000401:11297) [ 5215.965280] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11221 to 0x2c0000401:11265) [ 5217.246972] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5217.845707] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5341.733656] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5352.418967] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 5353.028779] Lustre: Failing over lustre-MDT0000 [ 5353.321588] Lustre: server umount lustre-MDT0000 complete [ 5367.287094] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5367.288488] LDISKFS-fs (dm-0): recovery complete [ 5367.291365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5368.728849] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5376.119068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13314 to 0x280000401:13345) [ 5376.120068] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13282 to 0x2c0000401:13313) [ 5377.374227] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5378.005601] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5381.817045] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 02:37:01 (1775889421) [ 5387.642280] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5389.143337] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5389.778962] Lustre: Failing over lustre-OST0000 [ 5389.831249] Lustre: server umount lustre-OST0000 complete [ 5403.722190] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5403.724080] LDISKFS-fs (dm-2): recovery complete [ 5403.727598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5405.612145] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5408.951880] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5409.564525] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5418.398391] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5419.891860] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5420.502333] Lustre: Failing over lustre-OST0000 [ 5420.545568] Lustre: server umount lustre-OST0000 complete [ 5434.435716] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5434.437138] LDISKFS-fs (dm-2): recovery complete [ 5434.440549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5436.366932] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5439.700108] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5440.333697] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5445.854558] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 02:38:05 (1775889485) [ 5568.445243] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5570.792077] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5581.496989] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 5582.098832] Lustre: Failing over lustre-MDT0001 [ 5582.102067] LustreError: lustre-MDT0001-osp-MDT0000: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 5582.104706] LustreError: Skipped 6 previous similar messages [ 5582.106101] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.110066] Lustre: Skipped 19 previous similar messages [ 5582.111298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5582.113021] Lustre: Skipped 8 previous similar messages [ 5582.328993] Lustre: server umount lustre-MDT0001 complete [ 5583.328078] LustreError: 8411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.333565] LustreError: 8411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 72 previous similar messages [ 5583.464100] Lustre: Failing over lustre-MDT0000 [ 5583.652521] Lustre: server umount lustre-MDT0000 complete [ 5598.257059] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5598.260618] LDISKFS-fs (dm-1): recovery complete [ 5598.266847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.310626] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5598.312964] LDISKFS-fs (dm-0): recovery complete [ 5598.317723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.361738] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5598.365538] LustreError: Skipped 3 previous similar messages [ 5598.368393] LustreError: 191503:0:(llog.c:1646:llog_backup()) MGC192.168.204.130@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5598.368537] LustreError: 191502:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGC192.168.204.130@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff94b747e30a00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5598.372367] Lustre: 191503:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.130@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5598.462114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5598.464370] Lustre: Skipped 5 previous similar messages [ 5598.471906] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5598.475109] Lustre: Skipped 5 previous similar messages [ 5599.526298] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5599.532069] Lustre: Skipped 6 previous similar messages [ 5599.883078] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5599.960796] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5603.680651] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5603.683234] Lustre: Skipped 19 previous similar messages [ 5604.648046] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5604.652159] Lustre: Skipped 5 previous similar messages [ 5604.666505] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5313 to 0x2c0000400:5345) [ 5604.667197] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5345 to 0x280000400:5377) [ 5610.269688] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13376 to 0x2c0000401:13409) [ 5610.269807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13409 to 0x280000401:13441) [ 5611.614293] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5612.264502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5612.817229] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5736.908918] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5739.317694] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5749.999663] Lustre: DEBUG MARKER: fail mds2 mds1 2 times [ 5750.613875] Lustre: Failing over lustre-MDT0001 [ 5750.743762] Lustre: server umount lustre-MDT0001 complete [ 5751.648092] Lustre: 3657:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775889634/real 1775889634] req@ffff94b6498fea00 x1862149541882112/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775889650 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5751.655296] Lustre: 3657:0:(client.c:2482:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5751.866125] Lustre: Failing over lustre-MDT0000 [ 5752.069994] Lustre: server umount lustre-MDT0000 complete [ 5766.346542] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5766.349804] LDISKFS-fs (dm-1): recovery complete [ 5766.358225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.426184] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5766.427621] LDISKFS-fs (dm-0): recovery complete [ 5766.431038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.469231] LustreError: 194747:0:(llog.c:1646:llog_backup()) MGC192.168.204.130@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5766.469325] LustreError: 194748:0:(ldlm_resource.c:1170:ldlm_resource_complain()) MGC192.168.204.130@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff94b77d7eee00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5766.472040] Lustre: 194747:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.130@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5767.830397] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5767.869074] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5769.567123] Lustre: 3658:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775889794/real 1775889794] req@ffff94b77da39880 x1862149553229696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775889810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5769.575506] Lustre: 3658:0:(client.c:2482:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5771.982030] LustreError: 174452:0:(ldlm_lockd.c:915:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-lustre-MDT0001_UUID lock: ffff94b64c33f400/0xff426f1a0a079d8b lrc: 3/0,0 mode: PW/PW res: [0x240001f58:0x10c1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x40000000000020 nid: 0@lo remote: 0xff426f1a0a079d84 expref: 57 pid: 174453 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5772.722963] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5345 to 0x280000400:5409) [ 5772.722977] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5313 to 0x2c0000400:5377) [ 5777.579047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13409 to 0x280000401:13473) [ 5777.579163] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13376 to 0x2c0000401:13441) [ 5778.799989] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5779.362472] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5779.906139] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5782.176078] Lustre: 3655:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775889804/real 1775889804] req@ffff94b77da38380 x1862149553230848/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775889820 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5782.187090] Lustre: 3655:0:(client.c:2482:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5783.271929] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 02:43:42 (1775889822) [ 5785.708241] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5786.576217] Lustre: Failing over lustre-MDT0000 [ 5786.735552] Lustre: server umount lustre-MDT0000 complete [ 5800.349252] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5800.350307] LDISKFS-fs (dm-0): recovery complete [ 5800.353100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.575124] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5804.326879] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5820.716323] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5820.727475] Lustre: 195520:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b7484e6680 x1862149580877440/t356482286205(356482286205) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:12/0 lens 520/3488 e 0 to 0 dl 1775889872 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5820.766755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13505) [ 5820.766769] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13376 to 0x2c0000401:13473) [ 5821.923167] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5822.391623] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5825.457540] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 02:44:25 (1775889865) [ 5827.914572] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5828.814842] Lustre: Failing over lustre-MDT0000 [ 5828.987680] Lustre: server umount lustre-MDT0000 complete [ 5843.004378] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5843.006099] LDISKFS-fs (dm-0): recovery complete [ 5843.009090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.403535] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5845.294813] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5845.296339] LustreError: 194755:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b64b803480 x1862149580877440/t356482286205(356482286205) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:37/0 lens 520/664 e 0 to 0 dl 1775889897 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5861.680850] Lustre: lustre-MDT0000: Client e08df21d-d694-4084-80c8-4b7c9f2345cf (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5861.686266] Lustre: 195520:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b64cb63b80 x1862149580877440/t356482286205(356482286205) o101->e08df21d-d694-4084-80c8-4b7c9f2345cf@192.168.204.30@tcp:53/0 lens 520/3488 e 0 to 0 dl 1775889913 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5861.716194] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13475 to 0x2c0000401:13505) [ 5861.716367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13537) [ 5863.054163] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5863.565939] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5867.026083] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 02:45:06 (1775889906) [ 5868.821337] Lustre: Failing over lustre-OST0000 [ 5868.872720] Lustre: server umount lustre-OST0000 complete [ 5870.068700] Lustre: Failing over lustre-MDT0000 [ 5870.157458] Lustre: server umount lustre-MDT0000 complete [ 5882.931640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.344048] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5885.068118] Lustre: lustre-MDT0000: Denying connection for new client 1cca2775-cb54-4781-ad17-a226beab9d6c (at 192.168.204.30@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5888.500051] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13475 to 0x2c0000401:13537) [ 5892.260189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5893.813568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13569) [ 5894.187173] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5898.195142] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 02:45:37 (1775889937) [ 5898.535213] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5898.537777] LustreError: 194761:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b77f7f0380 x1862149554730752/t377957122055(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:90/0 lens 1312/4320 e 0 to 0 dl 1775889950 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5901.014113] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5901.669362] Lustre: Failing over lustre-MDT0000 [ 5901.926724] Lustre: server umount lustre-MDT0000 complete [ 5914.079224] Lustre: 194781:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775889939/real 1775889939] req@ffff94b77f7f0e00 x1862149554730752/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1775889955 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5914.087430] Lustre: 194781:0:(client.c:2482:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 5915.910618] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5915.912385] LDISKFS-fs (dm-0): recovery complete [ 5915.915314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5917.177575] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5921.266206] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13569) [ 5921.266250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13601) [ 5922.237185] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5922.699103] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5925.874526] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 02:46:05 (1775889965) [ 5928.378295] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5930.637277] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5931.202755] Lustre: Failing over lustre-MDT0001 [ 5931.205704] LustreError: 204533:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x20002b381:0x4:0x0].0x0 (ffff94b6410e7c00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5937.222893] Lustre: server umount lustre-MDT0001 complete [ 5951.175088] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5951.176310] LDISKFS-fs (dm-1): recovery complete [ 5951.180150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5952.569366] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5956.592564] Lustre: 195520:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b74870ca80 x1862149595678080/t21474839171(0) o36->1cca2775-cb54-4781-ad17-a226beab9d6c@192.168.204.30@tcp:170/0 lens 560/2880 e 0 to 0 dl 1775890030 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5956.597793] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5388 to 0x2c0000400:5409) [ 5956.597811] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5420 to 0x280000400:5441) [ 5957.738664] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5958.265610] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5961.758689] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 02:46:41 (1775890001) [ 5962.106533] LustreError: 194760:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b7770a7b80 x1862149554791168/t382252089377(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:154/0 lens 2520/4320 e 0 to 0 dl 1775890014 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5962.114066] LustreError: 194760:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 5964.582723] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5967.004546] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5967.613126] Lustre: Failing over lustre-MDT0000 [ 5967.861358] Lustre: server umount lustre-MDT0000 complete [ 5978.591153] Lustre: 205158:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775890003/real 1775890003] req@ffff94b64c523480 x1862149554791168/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1775890019 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5981.853353] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5981.855058] LDISKFS-fs (dm-0): recovery complete [ 5981.858985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.312370] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5987.339414] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13601) [ 5987.339414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13633) [ 5988.495463] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5989.058959] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5991.011531] Lustre: Failing over lustre-MDT0001 [ 5991.101653] Lustre: server umount lustre-MDT0001 complete [ 6005.250946] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6005.252093] LDISKFS-fs (dm-1): recovery complete [ 6005.255773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6006.550596] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6010.865617] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5452 to 0x280000400:5473) [ 6010.865624] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5420 to 0x2c0000400:5441) [ 6011.893746] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6012.385937] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6015.756191] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 02:47:35 (1775890055) [ 6016.065500] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6016.066840] Lustre: Skipped 2 previous similar messages [ 6021.329097] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6023.644109] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6024.258799] Lustre: Failing over lustre-MDT0000 [ 6024.403794] Lustre: server umount lustre-MDT0000 complete [ 6025.511638] LustreError: 145406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775890066 with bad export cookie 18393385985762707841 [ 6025.513110] Lustre: Failing over lustre-MDT0001 [ 6025.515172] LustreError: 145406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6030.815563] LustreError: 210467:0:(client.c:1383:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff94b644779c00 x1862149554851712/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6030.823056] LustreError: 210467:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6030.926339] Lustre: server umount lustre-MDT0001 complete [ 6045.255958] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6045.257497] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6045.257729] LDISKFS-fs (dm-0): recovery complete [ 6045.258834] LDISKFS-fs (dm-1): recovery complete [ 6045.263217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6045.265420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6045.331053] LustreError: 211567:0:(llog.c:1646:llog_backup()) MGC192.168.204.130@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6045.333331] Lustre: 211567:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.130@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6050.783373] LustreError: 3654:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b744791f80 x1862149554852864/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6052.329906] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6052.442889] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6056.125614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13665) [ 6056.125626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13633) [ 6056.145625] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5452 to 0x2c0000400:5473) [ 6056.146039] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5484 to 0x280000400:5505) [ 6056.149210] Lustre: 211577:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b7487c6d80 x1862149595744000/t30064771140(0) o36->1cca2775-cb54-4781-ad17-a226beab9d6c@192.168.204.30@tcp:272/0 lens 560/2880 e 0 to 0 dl 1775890132 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6057.421530] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6058.033674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6058.525402] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6061.930178] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 02:48:21 (1775890101) [ 6062.279869] LustreError: 211618:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b648fa8700 x1862149595776768/t34359738436(0) o36->1cca2775-cb54-4781-ad17-a226beab9d6c@192.168.204.30@tcp:254/0 lens 560/448 e 0 to 0 dl 1775890114 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6062.286865] LustreError: 211618:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 6067.765326] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6068.383115] Lustre: Failing over lustre-MDT0000 [ 6068.643888] Lustre: server umount lustre-MDT0000 complete [ 6078.769222] Lustre: lustre-MDT0001: Client 1cca2775-cb54-4781-ad17-a226beab9d6c (at 192.168.204.30@tcp) reconnecting [ 6078.773602] Lustre: Skipped 2 previous similar messages [ 6082.527571] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6082.528886] LDISKFS-fs (dm-0): recovery complete [ 6082.531533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6083.816062] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6087.677805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13697) [ 6087.677858] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13665) [ 6088.744925] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6089.251479] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6092.652967] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 02:48:52 (1775890132) [ 6095.300078] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6095.899727] Lustre: Failing over lustre-MDT0001 [ 6095.982626] Lustre: server umount lustre-MDT0001 complete [ 6110.004993] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6110.007327] LDISKFS-fs (dm-1): recovery complete [ 6110.015090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.450670] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6115.308931] Lustre: 211576:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b6455bca80 x1862149595807488/t34359738505(0) o36->1cca2775-cb54-4781-ad17-a226beab9d6c@192.168.204.30@tcp:307/0 lens 560/2880 e 0 to 0 dl 1775890167 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6115.315437] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5494 to 0x2c0000400:5537) [ 6115.315447] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5526 to 0x280000400:5569) [ 6115.318803] Lustre: 211576:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6116.484467] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6117.020657] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6120.650318] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 02:49:20 (1775890160) [ 6126.522463] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6129.007318] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6129.599070] Lustre: Failing over lustre-MDT0000 [ 6129.874186] Lustre: server umount lustre-MDT0000 complete [ 6144.019626] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6144.022219] LDISKFS-fs (dm-0): recovery complete [ 6144.026800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.425629] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6149.628279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13729) [ 6149.628465] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13697) [ 6150.732502] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6151.249910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6153.144933] Lustre: Failing over lustre-MDT0001 [ 6153.231502] Lustre: server umount lustre-MDT0001 complete [ 6167.272582] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6167.273964] LDISKFS-fs (dm-1): recovery complete [ 6167.280667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.672254] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6172.663928] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5548 to 0x2c0000400:5569) [ 6172.663960] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5580 to 0x280000400:5601) [ 6173.754129] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6174.258993] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6177.723519] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 02:50:17 (1775890217) [ 6178.067930] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6178.070268] Lustre: Skipped 3 previous similar messages [ 6183.453033] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6185.854377] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6186.487304] Lustre: Failing over lustre-MDT0000 [ 6186.797590] Lustre: server umount lustre-MDT0000 complete [ 6187.970614] LustreError: 42168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775890228 with bad export cookie 18393385985762721533 [ 6187.972224] Lustre: Failing over lustre-MDT0001 [ 6187.974259] LustreError: 42168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6188.000551] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6188.000993] LustreError: 211618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.001184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6188.001190] Lustre: Skipped 17 previous similar messages [ 6188.004983] Lustre: Skipped 57 previous similar messages [ 6188.014111] LustreError: 211618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 162 previous similar messages [ 6188.088660] Lustre: server umount lustre-MDT0001 complete [ 6202.657569] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6202.657624] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6202.659307] LDISKFS-fs (dm-1): recovery complete [ 6202.660862] LDISKFS-fs (dm-0): recovery complete [ 6202.662644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6202.667170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6202.750582] LustreError: 222160:0:(llog.c:1646:llog_backup()) MGC192.168.204.130@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6202.754829] Lustre: 222160:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.130@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6212.575361] LustreError: 3654:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b7484e6300 x1862149555001728/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6212.659184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6212.662637] Lustre: Skipped 17 previous similar messages [ 6212.670214] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6212.673505] Lustre: Skipped 17 previous similar messages [ 6212.736795] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 6212.742495] LustreError: Skipped 14 previous similar messages [ 6214.071457] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6214.074199] Lustre: Skipped 16 previous similar messages [ 6214.082049] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6214.184991] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6218.209940] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6218.213388] Lustre: Skipped 58 previous similar messages [ 6218.855463] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6218.858841] Lustre: Skipped 17 previous similar messages [ 6218.864082] Lustre: 222175:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b7484e7100 x1862149595872128/t42949673028(0) o36->1cca2775-cb54-4781-ad17-a226beab9d6c@192.168.204.30@tcp:410/0 lens 560/2880 e 0 to 0 dl 1775890270 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6218.872191] Lustre: 222175:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6218.921829] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5580 to 0x2c0000400:5601) [ 6218.921829] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5612 to 0x280000400:5633) [ 6218.932617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13761) [ 6218.932617] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13729) [ 6220.083123] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6220.616298] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6221.109914] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6224.578453] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 02:51:04 (1775890264) [ 6224.937428] LustreError: 174453:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b74874ce00 x1862149555028864/t403726925838(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:416/0 lens 1728/4320 e 0 to 0 dl 1775890276 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6224.943935] LustreError: 174453:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 3 previous similar messages [ 6227.566457] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6228.183864] Lustre: Failing over lustre-MDT0001 [ 6234.172072] Lustre: server umount lustre-MDT0001 complete [ 6248.431905] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6248.433226] LDISKFS-fs (dm-1): recovery complete [ 6248.437612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6249.794215] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6254.069063] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5612 to 0x2c0000400:5633) [ 6254.069437] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5644 to 0x280000400:5665) [ 6255.247044] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6255.768856] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6259.044740] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 02:51:38 (1775890298) [ 6261.758685] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6262.388249] Lustre: Failing over lustre-MDT0000 [ 6262.646266] Lustre: server umount lustre-MDT0000 complete [ 6275.551130] Lustre: 224470:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775890300/real 1775890300] req@ffff94b648fabb80 x1862149555059456/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1488/4320 e 0 to 1 dl 1775890316 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6276.564997] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6276.566208] LDISKFS-fs (dm-0): recovery complete [ 6276.570771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.605511] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6276.608920] LustreError: Skipped 10 previous similar messages [ 6277.974229] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6281.729384] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13761) [ 6281.729414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13793) [ 6282.861942] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6283.441993] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6286.972582] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 02:52:06 (1775890326) [ 6289.697989] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6292.186541] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6292.822439] Lustre: Failing over lustre-MDT0000 [ 6293.057479] Lustre: server umount lustre-MDT0000 complete [ 6303.711110] Lustre: 224470:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775890328/real 1775890328] req@ffff94b776c43b80 x1862149555085824/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1744/4320 e 0 to 1 dl 1775890344 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6307.041445] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6307.042791] LDISKFS-fs (dm-0): recovery complete [ 6307.045365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6308.449128] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6312.453999] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13825) [ 6312.454077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13793) [ 6313.575282] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6314.080834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6315.994329] Lustre: Failing over lustre-MDT0001 [ 6316.083305] Lustre: server umount lustre-MDT0001 complete [ 6330.585118] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6330.587694] LDISKFS-fs (dm-1): recovery complete [ 6330.593352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.969840] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6335.986200] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5644 to 0x280000400:5697) [ 6335.986209] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5612 to 0x2c0000400:5665) [ 6337.144308] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6337.671653] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6341.201180] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 02:53:00 (1775890380) [ 6344.053158] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6346.482554] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6347.204944] Lustre: Failing over lustre-MDT0000 [ 6347.533696] Lustre: server umount lustre-MDT0000 complete [ 6348.974633] LustreError: 145406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775890389 with bad export cookie 18393385985762731956 [ 6348.976209] Lustre: Failing over lustre-MDT0001 [ 6348.980177] LustreError: 145406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6348.983436] LustreError: 231661:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x20002b381:0x1f:0x0].0x0 (ffff94b74ffc0600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6354.911928] LustreError: 231661:0:(client.c:1383:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff94b74fcd4e00 x1862149555137152/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6354.921328] LustreError: 231661:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6355.039766] Lustre: server umount lustre-MDT0001 complete [ 6370.161906] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6370.163622] LDISKFS-fs (dm-0): recovery complete [ 6370.165122] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6370.167345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6370.168225] LDISKFS-fs (dm-1): recovery complete [ 6370.173065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.855603] LustreError: 3654:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b77d6eb800 x1862149555137920/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6375.364494] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6375.468886] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6379.248895] Lustre: 232778:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b648acb800 x1862149595948800/t55834574850(0) o36->1cca2775-cb54-4781-ad17-a226beab9d6c@192.168.204.30@tcp:596/0 lens 496/2888 e 0 to 0 dl 1775890456 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6379.254923] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5612 to 0x2c0000400:5697) [ 6379.254949] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5644 to 0x280000400:5729) [ 6379.258259] Lustre: 232778:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6392.157218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13825) [ 6392.157265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13857) [ 6393.660709] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6394.187909] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6394.666696] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6398.213559] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 02:53:57 (1775890437) [ 6401.240244] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6401.986474] Lustre: Failing over lustre-MDT0000 [ 6402.182120] Lustre: server umount lustre-MDT0000 complete [ 6414.118106] Lustre: lustre-MDT0001: Client 1cca2775-cb54-4781-ad17-a226beab9d6c (at 192.168.204.30@tcp) reconnecting [ 6414.121560] Lustre: Skipped 1 previous similar message [ 6416.298099] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6416.299466] LDISKFS-fs (dm-0): recovery complete [ 6416.302385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6417.794087] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6421.503076] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13857) [ 6421.503089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13889) [ 6422.572029] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6423.134778] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6426.781426] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 02:54:26 (1775890466) [ 6429.622145] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6430.228388] Lustre: Failing over lustre-MDT0001 [ 6430.315039] Lustre: server umount lustre-MDT0001 complete [ 6444.187154] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6444.188606] LDISKFS-fs (dm-1): recovery complete [ 6444.192395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6445.744570] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6449.653464] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5612 to 0x2c0000400:5729) [ 6449.653467] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5644 to 0x280000400:5761) [ 6450.708129] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6451.176686] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6454.581116] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 02:54:54 (1775890494) [ 6454.941498] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6454.942942] Lustre: Skipped 6 previous similar messages [ 6457.683028] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6460.156758] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6460.795566] Lustre: Failing over lustre-MDT0000 [ 6460.940690] Lustre: server umount lustre-MDT0000 complete [ 6474.798947] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6474.800178] LDISKFS-fs (dm-0): recovery complete [ 6474.802778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6476.256411] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6480.379382] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13889) [ 6480.379394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13921) [ 6481.468373] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6481.993745] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6483.898901] Lustre: Failing over lustre-MDT0001 [ 6483.980867] Lustre: server umount lustre-MDT0001 complete [ 6497.971034] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6497.972351] LDISKFS-fs (dm-1): recovery complete [ 6497.976240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6499.300657] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6503.412671] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5644 to 0x280000400:5793) [ 6503.412695] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5612 to 0x2c0000400:5761) [ 6504.512578] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6505.059908] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6508.423120] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 02:55:48 (1775890548) [ 6508.779786] LustreError: 232778:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff94b64c520a80 x1862149596004224/t68719476738(0) o36->1cca2775-cb54-4781-ad17-a226beab9d6c@192.168.204.30@tcp:700/0 lens 496/456 e 0 to 0 dl 1775890560 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6508.788278] LustreError: 232778:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 6 previous similar messages [ 6511.627945] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6515.121277] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6515.794050] Lustre: Failing over lustre-MDT0000 [ 6516.026019] Lustre: server umount lustre-MDT0000 complete [ 6517.196644] LustreError: 145408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775890558 with bad export cookie 18393385985762739516 [ 6517.198125] Lustre: Failing over lustre-MDT0001 [ 6517.200722] LustreError: 145408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6517.306841] Lustre: server umount lustre-MDT0001 complete [ 6531.973128] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6531.976588] LDISKFS-fs (dm-0): recovery complete [ 6531.977510] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6531.980965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6531.981747] LDISKFS-fs (dm-1): recovery complete [ 6531.992745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.623213] Lustre: 3657:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775890559/real 1775890559] req@ffff94b7484e6d80 x1862149555271552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775890575 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6534.639029] Lustre: 3657:0:(client.c:2482:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6542.815363] LustreError: 3654:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b64a10d500 x1862149555274368/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6544.221855] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6544.319745] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6546.697603] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13539 to 0x2c0000401:13921) [ 6546.697620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13475 to 0x280000401:13953) [ 6549.751927] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5612 to 0x2c0000400:5793) [ 6549.751958] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5644 to 0x280000400:5825) [ 6550.862243] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6551.374812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6551.900869] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6555.856373] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 02:56:35 (1775890595) [ 6556.617444] Lustre: 244661:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 1cca2775-cb54-4781-ad17-a226beab9d6c at adminstrative request [ 6561.392751] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 02:56:41 (1775890601) [ 6564.053469] Lustre: Failing over lustre-MDT0000 [ 6564.307483] Lustre: server umount lustre-MDT0000 complete [ 6577.863132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6579.500708] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6583.347930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13973 to 0x2c0000401:14017) [ 6583.347970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14004 to 0x280000401:14049) [ 6584.503319] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6585.026765] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6588.484693] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 02:57:08 (1775890628) [ 6592.042096] Lustre: Failing over lustre-OST0000 [ 6592.119372] Lustre: server umount lustre-OST0000 complete [ 6605.360975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6607.649161] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6610.719447] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6611.247181] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6614.987150] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 02:57:34 (1775890654) [ 6616.623493] Lustre: Failing over lustre-MDT0000 [ 6616.784306] Lustre: server umount lustre-MDT0000 complete [ 6618.083385] LustreError: 145409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775890659 with bad export cookie 18393385985762752116 [ 6619.655027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.274607] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6622.289308] Lustre: lustre-MDT0000: Denying connection for new client 69fb1825-4d63-4389-9d87-dd3f2c3e7147 (at 192.168.204.30@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 6622.293502] Lustre: Skipped 2 previous similar messages [ 6625.264364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14004 to 0x280000401:14081) [ 6625.264390] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13973 to 0x2c0000401:14049) [ 6629.547977] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 02:57:49 (1775890669) [ 6632.119356] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6632.890931] Lustre: Failing over lustre-OST0000 [ 6632.924754] Lustre: server umount lustre-OST0000 complete [ 6647.703433] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6647.707961] LDISKFS-fs (dm-2): recovery complete [ 6647.713067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6650.494935] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6654.489422] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6655.302728] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6659.811146] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 02:58:19 (1775890699) [ 6663.735400] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6665.916985] Lustre: Failing over lustre-OST0000 [ 6665.965939] Lustre: server umount lustre-OST0000 complete [ 6680.579659] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6680.583053] LDISKFS-fs (dm-2): recovery complete [ 6680.590230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6682.261853] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.30@tcp inode [0x2000320e1:0x5:0x0] object 0x280000401:14083 extent [0-4194303]: client csum fd0012a7, server csum 5ca712b3 [ 6683.042681] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6686.928749] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6687.562305] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6691.565797] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 02:58:51 (1775890731) [ 6694.314493] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6697.279982] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6699.683076] Lustre: Failing over lustre-MDT0000 [ 6699.967961] Lustre: server umount lustre-MDT0000 complete [ 6701.502674] LustreError: 8401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775890742 with bad export cookie 18393385985762766347 [ 6701.505250] Lustre: Failing over lustre-OST0000 [ 6701.507970] LustreError: 8401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6701.565408] Lustre: server umount lustre-OST0000 complete [ 6717.059783] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6717.062502] LDISKFS-fs (dm-0): recovery complete [ 6717.068471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.111459] LustreError: 3654:0:(client.c:1393:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b7493fce00 x1862149555423360/t0(0) o250->MGC192.168.204.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6727.705296] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6731.373357] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13973 to 0x2c0000401:14081) [ 6742.324452] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6742.326909] LDISKFS-fs (dm-2): recovery complete [ 6742.334992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6744.781710] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6751.502827] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 02:59:51 (1775890791) [ 6757.069561] Lustre: Failing over lustre-OST0000 [ 6757.150500] Lustre: server umount lustre-OST0000 complete [ 6758.874563] Lustre: Failing over lustre-MDT0000 [ 6759.217980] Lustre: server umount lustre-MDT0000 complete [ 6773.162487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.887856] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6778.371978] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13973 to 0x2c0000401:14113) [ 6782.152377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6784.711279] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6785.711788] Lustre: lustre-OST0000: Denying connection for new client 09805aeb-f149-4e9d-b968-cfe63a25c87d (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 6785.722703] Lustre: Skipped 1 previous similar message [ 6790.949898] Lustre: lustre-OST0000: Denying connection for new client 09805aeb-f149-4e9d-b968-cfe63a25c87d (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 6790.957487] Lustre: Skipped 1 previous similar message [ 6801.190049] Lustre: lustre-OST0000: Denying connection for new client 09805aeb-f149-4e9d-b968-cfe63a25c87d (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 6801.195425] Lustre: Skipped 1 previous similar message [ 6821.670734] Lustre: lustre-OST0000: Denying connection for new client 09805aeb-f149-4e9d-b968-cfe63a25c87d (at 192.168.204.30@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:31 [ 6821.681693] Lustre: Skipped 3 previous similar messages [ 6853.500177] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6853.505058] Lustre: 257719:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 69fb1825-4d63-4389-9d87-dd3f2c3e7147@ [ 6853.513098] Lustre: 257719:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6853.518655] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6853.527377] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6853.527492] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 6853.529705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14132 to 0x280000401:14153) [ 6853.533647] Lustre: Skipped 67 previous similar messages [ 6853.540598] Lustre: Skipped 21 previous similar messages [ 6858.761223] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 6863.265142] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 6865.220563] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 03:01:44 (1775890904) [ 6866.600218] Lustre: Failing over lustre-OST0000 [ 6866.637396] Lustre: server umount lustre-OST0000 complete [ 6867.756673] LustreError: 177948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6867.766102] LustreError: 177948:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 228 previous similar messages [ 6868.959663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6868.959721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6868.963774] LustreError: Skipped 13 previous similar messages [ 6868.969343] Lustre: Skipped 59 previous similar messages [ 6879.716083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6879.773074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6879.776192] Lustre: Skipped 22 previous similar messages [ 6879.779798] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6879.781681] Lustre: Skipped 22 previous similar messages [ 6881.391019] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6881.393020] Lustre: Skipped 22 previous similar messages [ 6881.642619] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6885.600632] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 03:02:05 (1775890925) [ 6886.865668] Lustre: Failing over lustre-OST0000 [ 6886.879518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6886.881685] Lustre: Skipped 17 previous similar messages [ 6886.912172] Lustre: server umount lustre-OST0000 complete [ 6899.469273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6901.230873] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6901.485911] LustreError: 260705:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6901.488744] LustreError: 260705:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 78 previous similar messages [ 6907.871112] Lustre: *** cfs_fail_loc=715, val=40*** [ 6917.087115] Lustre: 3654:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775890942/real 1775890942] req@ffff94b7727a2a00 x1862149555532672/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775890958 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 6917.094126] Lustre: 3654:0:(client.c:2482:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 6917.096538] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [ 6917.099972] Lustre: Skipped 1 previous similar message [ 6917.871270] Lustre: lustre-OST0000: Client 09805aeb-f149-4e9d-b968-cfe63a25c87d (at 192.168.204.30@tcp) reconnected, waiting for 3 clients in recovery for 1:23 [ 6923.231093] Lustre: *** cfs_fail_loc=715, val=40*** [ 6923.232204] Lustre: Skipped 2 previous similar messages [ 6933.471537] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 6933.474110] Lustre: Skipped 1 previous similar message [ 6939.615145] Lustre: *** cfs_fail_loc=715, val=40*** [ 6939.617015] Lustre: Skipped 2 previous similar messages [ 6941.535106] LustreError: 260705:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6941.538484] LustreError: 260705:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 78 previous similar messages [ 6942.778449] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6943.264367] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6946.411632] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 03:03:06 (1775890986) [ 6947.580476] Lustre: Failing over lustre-MDT0000 [ 6947.782643] Lustre: server umount lustre-MDT0000 complete [ 6960.361774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.395859] LustreError: MGC192.168.204.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6960.398524] LustreError: Skipped 9 previous similar messages [ 6961.633884] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6965.735185] LustreError: 262143:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6971.359217] Lustre: *** cfs_fail_loc=715, val=80*** [ 6971.362807] Lustre: Skipped 2 previous similar messages [ 6981.087855] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6981.413861] Lustre: lustre-MDT0000: Client 09805aeb-f149-4e9d-b968-cfe63a25c87d (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 1:25 [ 6981.417369] Lustre: Skipped 1 previous similar message [ 6987.231194] Lustre: *** cfs_fail_loc=715, val=80*** [ 6987.233686] Lustre: Skipped 1 previous similar message [ 6997.471792] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6997.798541] Lustre: lustre-MDT0000: Client 09805aeb-f149-4e9d-b968-cfe63a25c87d (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 1:08 [ 7003.615168] Lustre: *** cfs_fail_loc=715, val=80*** [ 7003.618191] Lustre: Skipped 1 previous similar message [ 7013.158653] Lustre: lustre-MDT0000: Client 09805aeb-f149-4e9d-b968-cfe63a25c87d (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 7013.855558] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7019.999238] Lustre: *** cfs_fail_loc=715, val=80*** [ 7020.002013] Lustre: Skipped 2 previous similar messages [ 7029.215411] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7029.542083] Lustre: lustre-MDT0000: Client 09805aeb-f149-4e9d-b968-cfe63a25c87d (at 192.168.204.30@tcp) reconnected, waiting for 2 clients in recovery for 0:36 [ 7045.599209] Lustre: 3654:0:(client.c:2482:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775891070/real 1775891070] req@ffff94b7727a0700 x1862149555597696/t0(0) o400->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 1 dl 1775891086 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 7045.607204] Lustre: 3654:0:(client.c:2482:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 7045.609747] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7045.815080] LustreError: 262143:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7045.819928] Lustre: 262143:0:(ldlm_lib.c:2931:target_recovery_thread()) too long recovery - read logs [ 7045.822549] LustreError: dumping log to /tmp/lustre-log.1775891086.262143 [ 7045.880410] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14126 to 0x2c0000401:14145) [ 7045.880410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14167 to 0x280000401:14185) [ 7047.058292] Lustre: DEBUG MARKER: oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7047.546246] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7050.981639] Lustre: DEBUG MARKER: == replay-single test complete, duration 6813 sec ======== 03:04:50 (1775891090) [ 7051.562744] Lustre: DEBUG MARKER: === replay-single: start cleanup 03:04:51 (1775891091) === [ 7054.718505] Lustre: DEBUG MARKER: === replay-single: finish cleanup 03:04:54 (1775891094) === [ 7091.029932] Lustre: server umount lustre-MDT0000 complete [ 7094.332517] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775891135 with bad export cookie 18393385985762779101 [ 7094.340063] LustreError: 6494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7094.472931] Lustre: server umount lustre-MDT0001 complete [ 7107.268944] Lustre: server umount lustre-OST0000 complete [ 7119.358679] Lustre: server umount lustre-OST0001 complete [ 7124.488021] Lustre: DEBUG MARKER: oleg430-server.virtnet: executing unload_modules_local [ 7125.722885] Key type lgssc unregistered [ 7125.895532] LNet: 265309:0:(lib-ptl.c:970:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7125.901058] LNetError: 265309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7125.910469] LNet: Removed LNI 192.168.204.130@tcp [ 7126.247127] Key type .llcrypt unregistered [ 7126.248381] Key type ._llcrypt unregistered