[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488834892 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003366] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008499] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010012] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011013] pid_max: default: 32768 minimum: 301 [ 0.012119] LSM: Security Framework initializing [ 0.014045] Yama: becoming mindful. [ 0.015040] SELinux: Initializing. [ 0.016080] *** VALIDATE selinux *** [ 0.025357] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029826] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033110] *** VALIDATE tmpfs *** [ 0.034484] *** VALIDATE proc *** [ 0.036263] *** VALIDATE cgroup *** [ 0.037013] *** VALIDATE cgroup2 *** [ 0.039242] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040161] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043011] Spectre V2 : User space: Vulnerable [ 0.044010] Speculative Store Bypass: Vulnerable [ 0.047272] debug: unmapping init [mem 0xffffffff86859000-0xffffffff86860fff] [ 0.049170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050595] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051026] ... version: 2 [ 0.052013] ... bit width: 48 [ 0.053019] ... generic registers: 4 [ 0.054012] ... value mask: 0000ffffffffffff [ 0.055012] ... max period: 00007fffffffffff [ 0.056012] ... fixed-purpose events: 3 [ 0.057011] ... event mask: 000000070000000f [ 0.058353] rcu: Hierarchical SRCU implementation. [ 0.060439] smp: Bringing up secondary CPUs ... [ 0.061589] x86: Booting SMP configuration: [ 0.062029] .... node #0, CPUs: #1 #2 #3 [ 0.065200] smp: Brought up 1 node, 4 CPUs [ 0.067018] smpboot: Max logical packages: 1 [ 0.068023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150023] node 0 deferred pages initialised in 78ms [ 0.154133] devtmpfs: initialized [ 0.155273] x86/mm: Memory block size: 128MB [ 0.158780] gcov: version magic: 0x41383552 [ 0.162312] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.166102] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168379] pinctrl core: initialized pinctrl subsystem [ 0.171192] [ 0.171805] ************************************************************* [ 0.174014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176011] ** ** [ 0.178012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180014] ** ** [ 0.183012] ** This means that this kernel is built to expose internal ** [ 0.185011] ** IOMMU data structures, which may compromise security on ** [ 0.187017] ** your system. ** [ 0.189012] ** ** [ 0.191014] ** If you see this message and you are not debugging the ** [ 0.194015] ** kernel, report this immediately to your vendor! ** [ 0.196013] ** ** [ 0.198013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.201014] ************************************************************* [ 0.203897] NET: Registered protocol family 16 [ 0.204442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.210068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.214068] cpuidle: using governor menu [ 0.215742] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217441] PCI: Using configuration type 1 for base access [ 0.220123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.228125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.231106] cryptd: max_cpu_qlen set to 1000 [ 0.233154] ACPI: Added _OSI(Module Device) [ 0.235013] ACPI: Added _OSI(Processor Device) [ 0.236011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.238013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242336] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248399] ACPI: Interpreter enabled [ 0.250071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251014] ACPI: Using IOAPIC for interrupt routing [ 0.253119] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256415] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.264885] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.270020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.273077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278219] acpiphp: Slot [2] registered [ 0.279120] acpiphp: Slot [3] registered [ 0.281136] acpiphp: Slot [4] registered [ 0.282112] acpiphp: Slot [5] registered [ 0.284131] acpiphp: Slot [6] registered [ 0.286110] acpiphp: Slot [7] registered [ 0.287110] acpiphp: Slot [8] registered [ 0.288116] acpiphp: Slot [9] registered [ 0.290135] acpiphp: Slot [10] registered [ 0.291105] acpiphp: Slot [11] registered [ 0.293101] acpiphp: Slot [12] registered [ 0.294110] acpiphp: Slot [13] registered [ 0.296100] acpiphp: Slot [14] registered [ 0.297117] acpiphp: Slot [15] registered [ 0.299109] acpiphp: Slot [16] registered [ 0.300116] acpiphp: Slot [17] registered [ 0.302100] acpiphp: Slot [18] registered [ 0.303092] acpiphp: Slot [19] registered [ 0.305308] acpiphp: Slot [20] registered [ 0.306108] acpiphp: Slot [21] registered [ 0.308094] acpiphp: Slot [22] registered [ 0.309127] acpiphp: Slot [23] registered [ 0.311114] acpiphp: Slot [24] registered [ 0.312091] acpiphp: Slot [25] registered [ 0.313112] acpiphp: Slot [26] registered [ 0.315103] acpiphp: Slot [27] registered [ 0.316090] acpiphp: Slot [28] registered [ 0.317103] acpiphp: Slot [29] registered [ 0.318079] acpiphp: Slot [30] registered [ 0.320097] acpiphp: Slot [31] registered [ 0.321050] PCI host bridge to bus 0000:00 [ 0.323022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330040] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.333023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.336027] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.338026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.345620] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351483] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.360740] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.366477] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.376503] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378791] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385151] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.389018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.403017] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.409020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.413665] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.420018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.424014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.442017] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.453188] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.459020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.465019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.494246] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.500014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.507027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.524019] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.534545] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.541019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.546019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.564018] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.575261] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.582027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.589029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.605030] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.617150] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.629024] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.635018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.652019] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.662344] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.666512] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.668424] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.671421] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.673314] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.678100] iommu: Default domain type: Passthrough [ 0.681255] SCSI subsystem initialized [ 0.682206] ACPI: bus type USB registered [ 0.684170] usbcore: registered new interface driver usbfs [ 0.686120] usbcore: registered new interface driver hub [ 0.688129] usbcore: registered new device driver usb [ 0.689219] pps_core: LinuxPPS API ver. 1 registered [ 0.691018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.694095] PTP clock support registered [ 0.697068] EDAC MC: Ver: 3.0.0 [ 0.699055] PCI: Using ACPI for IRQ routing [ 0.700642] NetLabel: Initializing [ 0.702015] NetLabel: domain hash size = 128 [ 0.704014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.706092] NetLabel: unlabeled traffic allowed by default [ 0.709045] vgaarb: loaded [ 0.710185] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.712019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.721358] clocksource: Switched to clocksource kvm-clock [ 0.825776] VFS: Disk quotas dquot_6.6.0 [ 0.827434] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.829135] *** VALIDATE ramfs *** [ 0.830482] *** VALIDATE hugetlbfs *** [ 0.832230] pnp: PnP ACPI init [ 0.835298] pnp: PnP ACPI: found 6 devices [ 0.855645] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.858977] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.861124] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.863121] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.865323] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.867543] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.870164] NET: Registered protocol family 2 [ 0.873274] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.880381] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.884100] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.889717] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.893531] TCP: Hash tables configured (established 65536 bind 65536) [ 0.896856] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.900488] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.903754] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.907079] NET: Registered protocol family 1 [ 0.910586] RPC: Registered named UNIX socket transport module. [ 0.913021] RPC: Registered udp transport module. [ 0.914883] RPC: Registered tcp transport module. [ 0.916286] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.918498] NET: Registered protocol family 44 [ 0.920466] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.922681] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.924533] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.927228] PCI: CLS 0 bytes, default 64 [ 0.929320] Unpacking initramfs... [ 2.406514] debug: unmapping init [mem 0xffff94dc3cc54000-0xffff94dc3ffbffff] [ 2.412170] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.414752] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.418862] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.928225] Initialise system trusted keyrings [ 2.930182] Key type blacklist registered [ 2.932536] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.941760] zbud: loaded [ 2.944924] *** VALIDATE nfs *** [ 2.946384] *** VALIDATE nfs4 *** [ 2.948344] pstore: using deflate compression [ 2.951888] Platform Keyring initialized [ 3.063230] NET: Registered protocol family 38 [ 3.067612] Key type asymmetric registered [ 3.069108] Asymmetric key parser 'x509' registered [ 3.071444] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.074779] io scheduler mq-deadline registered [ 3.076658] io scheduler kyber registered [ 3.078856] io scheduler bfq registered [ 3.081076] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.084650] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.088178] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.090813] ACPI: Power Button [PWRF] [ 3.183252] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.273906] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.465758] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.560345] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.751454] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.786496] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.830860] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.838585] Non-volatile memory driver v1.3 [ 3.840528] Linux agpgart interface v0.103 [ 3.873262] virtio_blk virtio1: [vda] 133624 512-byte logical blocks (68.4 MB/65.2 MiB) [ 3.877497] vda: detected capacity change from 0 to 68415488 [ 3.893923] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.896809] vdb: detected capacity change from 0 to 1073741824 [ 3.912476] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.916098] vdc: detected capacity change from 0 to 2621440000 [ 3.930755] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.934526] vdd: detected capacity change from 0 to 2621440000 [ 3.951521] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.954566] vde: detected capacity change from 0 to 4294967296 [ 3.969734] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.973039] vdf: detected capacity change from 0 to 4294967296 [ 3.979661] libphy: Fixed MDIO Bus: probed [ 3.986750] usbcore: registered new interface driver usbserial_generic [ 3.989378] usbserial: USB Serial support registered for generic [ 3.991967] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.996666] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.998648] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.001672] mousedev: PS/2 mouse device common for all mice [ 4.005523] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.006545] rtc_cmos 00:05: RTC can wake from S4 [ 4.014921] rtc_cmos 00:05: registered as rtc0 [ 4.015896] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.017850] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.023929] intel_pstate: CPU model not supported [ 4.028389] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.030036] hid: raw HID events driver (C) Jiri Kosina [ 4.034838] usbcore: registered new interface driver usbhid [ 4.037118] usbhid: USB HID core driver [ 4.038978] drop_monitor: Initializing network drop monitor service [ 4.041781] Initializing XFRM netlink socket [ 4.044187] NET: Registered protocol family 10 [ 4.047886] Segment Routing with IPv6 [ 4.049398] NET: Registered protocol family 17 [ 4.051231] mpls_gso: MPLS GSO support [ 4.057652] RAS: Correctable Errors collector initialized. [ 4.060079] AVX version of gcm_enc/dec engaged. [ 4.061727] AES CTR mode by8 optimization enabled [ 4.150818] sched_clock: Marking stable (4150758096, 0)->(5055141317, -904383221) [ 4.154055] registered taskstats version 1 [ 4.157625] Loading compiled-in X.509 certificates [ 4.159961] zswap: loaded using pool lzo/zbud [ 4.187511] Key type big_key registered [ 4.202052] Key type encrypted registered [ 4.203708] ima: No TPM chip found, activating TPM-bypass! [ 4.205168] ima: Allocated hash algorithm: sha1 [ 4.206521] ima: No architecture policies found [ 4.208277] evm: Initialising EVM extended attributes: [ 4.210128] evm: security.selinux [ 4.211097] evm: security.ima [ 4.211851] evm: security.capability [ 4.212819] evm: HMAC attrs: 0x1 [ 4.215145] rtc_cmos 00:05: setting system clock to 2025-11-14 00:50:49 UTC (1763081449) [ 4.221063] debug: unmapping init [mem 0xffffffff87803000-0xffffffff879fffff] [ 4.224031] debug: unmapping init [mem 0xffffffff86582000-0xffffffff86858fff] [ 4.236084] Write protecting the kernel read-only data: 28672k [ 4.239505] debug: unmapping init [mem 0xffffffff84c03000-0xffffffff84dfffff] [ 4.242348] debug: unmapping init [mem 0xffffffff85514000-0xffffffff855fffff] [ 4.276854] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.285572] systemd[1]: Detected virtualization kvm. [ 4.287878] systemd[1]: Detected architecture x86-64. [ 4.289055] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.315938] systemd[1]: No hostname configured. [ 4.317742] systemd[1]: Set hostname to . [ 4.319921] random: systemd: uninitialized urandom read (16 bytes read) [ 4.322505] systemd[1]: Initializing machine ID from random generator. [ 4.362550] random: ln: uninitialized urandom read (6 bytes read) [ 4.465814] random: systemd: uninitialized urandom read (16 bytes read) [ 4.468632] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.473742] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.480868] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.142407] device-mapper: uevent: version 1.0.3 [ 5.144959] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.908069] virtio_net virtio0 ens2: renamed from eth0 [ 5.912227] random: fast init done [ 5.994142] scsi host0: ata_piix [ 6.015836] scsi host1: ata_piix [ 6.017694] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.020141] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.711157] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.717187] random: crng init done [ 10.718802] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.143706] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.392185] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.678654] SELinux: Disabled at runtime. [ 12.744922] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.752777] systemd[1]: Detected virtualization kvm. [ 12.754513] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.297744] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.301396] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.309115] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.312480] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.323182] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.335780] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.346736] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ 13.537921] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.813897] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.601879] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.665986] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.089590] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.189933] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (6s / no limit)[ 20.489995] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (7s / no limit)[ 21.150838] NFS: Registering the id_resolver key type [ 21.159374] Key type id_resolver registered [ 21.161175] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 24.428030] hrtimer: interrupt took 2302381 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg405-server login: [ 67.264921] libcfs: loading out-of-tree module taints kernel. [ 67.291723] Key type ._llcrypt registered [ 67.293460] Key type .llcrypt registered [ 67.377721] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 82.882702] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 84.214835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.231178] alg: No test for adler32 (adler32-zlib) [ 85.493663] Lustre: Lustre: Build Version: 2.16.61_1_gaa570bc [ 86.397183] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 88.231205] Key type lgssc registered [ 89.924117] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.746667] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 108.983694] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.909017] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.093571] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 131.515324] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 145.716514] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 156.564590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 156.634233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.650710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.866040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.897834] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.958316] Lustre: lustre-MDT0000: new disk, initializing [ 158.049709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 158.070619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 160.992632] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.879707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.955614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 173.006260] Lustre: 6510:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.033857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.036477] Lustre: Skipped 1 previous similar message [ 173.111874] Lustre: lustre-MDT0001: new disk, initializing [ 173.166460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.192988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.200090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 176.172535] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.397516] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 187.859213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.921519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 188.119800] Lustre: lustre-OST0000: new disk, initializing [ 188.123278] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 188.161774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.314089] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.051687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 193.056526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 193.144055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.929498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.002538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 205.096423] Lustre: lustre-OST0001: new disk, initializing [ 205.106415] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 205.216918] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 210.868852] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.969816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 210.979613] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 211.028575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 223.141909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.798500] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.285942] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.024712] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing yml_node [ 251.056600] Lustre: DEBUG MARKER: Client: 2.16.61.1 [ 253.655592] Lustre: DEBUG MARKER: MDS: 2.16.61.1 [ 256.221688] Lustre: DEBUG MARKER: OSS: 2.16.61.1 [ 257.925465] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Thu Nov 13 19:55:01 EST 2025 [ 273.510747] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 275.253607] Lustre: DEBUG MARKER: === replay-single: start setup 19:55:19 (1763081719) === [ 278.274949] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing check_config_client /mnt/lustre [ 295.120283] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 297.811415] Lustre: 13186:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 300.478695] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 305.275640] Lustre: DEBUG MARKER: === replay-single: finish setup 19:55:49 (1763081749) === [ 306.878269] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 19:55:50 (1763081750) [ 313.419482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 315.281647] Lustre: Failing over lustre-MDT0000 [ 315.416900] LustreError: 14058:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 315.502724] Lustre: server umount lustre-MDT0000 complete [ 316.898035] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 316.906209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.431781] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.452532] Lustre: Skipped 2 previous similar messages [ 323.553354] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.574955] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 324.884126] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.675469] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.693021] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 333.793736] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.821932] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 334.816107] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763081763/real 1763081763] req@ffff94db888aa300 x1848724979557504/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763081779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 334.832535] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 335.819452] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 335.821208] LDISKFS-fs (dm-0): recovery complete [ 335.847040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 339.935756] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.948523] LustreError: 6521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 345.057195] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94dca051bb80 x1848724979566720/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 345.267105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 345.362937] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 349.533743] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 350.698095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 350.756720] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 356.042941] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 357.666189] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 365.352474] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 19:56:49 (1763081809) [ 366.943852] Lustre: Failing over lustre-OST0000 [ 366.974182] LustreError: 15496:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 366.977160] LustreError: 15496:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 367.049555] Lustre: server umount lustre-OST0000 complete [ 367.071951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 367.079794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.102592] LustreError: 8402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.134234] LustreError: 8402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 371.173684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 371.180355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 384.309400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 384.532634] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 386.237554] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 386.394141] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 386.394540] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 386.408776] Lustre: Skipped 3 previous similar messages [ 389.522774] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 396.028919] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 397.656845] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 408.106544] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 19:57:31 (1763081851) [ 415.659747] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 417.859558] Lustre: Failing over lustre-MDT0000 [ 418.006370] LustreError: 17199:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 418.009936] LustreError: 17199:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 418.097666] Lustre: server umount lustre-MDT0000 complete [ 420.320140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.323741] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 420.368452] Lustre: Skipped 3 previous similar messages [ 420.415721] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 436.703393] Lustre: 3651:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763081865/real 1763081865] req@ffff94dcbe8d1180 x1848724979616640/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763081881 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 436.714641] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 439.400216] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 439.402345] LDISKFS-fs (dm-0): recovery complete [ 439.413687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 446.944975] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94db88095c00 x1848724979625984/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 447.161617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.148877] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 451.830324] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 451.836576] Lustre: lustre-MDT0000: Denying connection for new client 05bc386e-41a5-4ad0-abd9-cd6612bc16b9 (at 192.168.204.5@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 452.577431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 452.583798] Lustre: Skipped 1 previous similar message [ 456.976348] Lustre: lustre-MDT0000: Denying connection for new client 05bc386e-41a5-4ad0-abd9-cd6612bc16b9 (at 192.168.204.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 462.097119] Lustre: lustre-MDT0000: Denying connection for new client 05bc386e-41a5-4ad0-abd9-cd6612bc16b9 (at 192.168.204.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 467.217789] Lustre: lustre-MDT0000: Denying connection for new client 05bc386e-41a5-4ad0-abd9-cd6612bc16b9 (at 192.168.204.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 472.337446] Lustre: lustre-MDT0000: Denying connection for new client 05bc386e-41a5-4ad0-abd9-cd6612bc16b9 (at 192.168.204.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 482.580300] Lustre: lustre-MDT0000: Denying connection for new client 05bc386e-41a5-4ad0-abd9-cd6612bc16b9 (at 192.168.204.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 482.599714] Lustre: Skipped 1 previous similar message [ 503.061171] Lustre: lustre-MDT0000: Denying connection for new client 05bc386e-41a5-4ad0-abd9-cd6612bc16b9 (at 192.168.204.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:17 [ 503.073854] Lustre: Skipped 3 previous similar messages [ 521.000267] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 521.002897] Lustre: 17852:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client e92f8891-7938-449c-ba1c-e64af6da43b5@ [ 521.019148] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 521.046699] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 521.046818] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 521.063324] Lustre: Skipped 2 previous similar messages [ 521.101559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 521.105711] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 532.990934] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 19:59:36 (1763081976) [ 540.455886] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 542.210843] Lustre: Failing over lustre-MDT0000 [ 542.329628] LustreError: 18911:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 542.332705] LustreError: 18911:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 542.430815] Lustre: server umount lustre-MDT0000 complete [ 544.737119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.742159] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.752647] Lustre: Skipped 2 previous similar messages [ 544.791662] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 561.119166] Lustre: 3651:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763081990/real 1763081990] req@ffff94dcbfb37100 x1848724979677440/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763082006 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 561.145821] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 562.071403] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 562.073734] LDISKFS-fs (dm-0): recovery complete [ 562.078901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.583839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.151379] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 576.999648] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 577.003776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 577.349232] Lustre: lustre-MDT0000: Denying connection for new client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 577.369078] Lustre: Skipped 3 previous similar messages [ 643.862916] Lustre: lustre-MDT0000: Denying connection for new client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:02 [ 643.873984] Lustre: Skipped 12 previous similar messages [ 646.002163] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 646.005286] Lustre: 19564:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 05bc386e-41a5-4ad0-abd9-cd6612bc16b9@ [ 646.015733] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 646.031650] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 646.033732] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 646.040890] Lustre: Skipped 2 previous similar messages [ 646.070223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 646.070322] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 657.041797] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 20:01:40 (1763082100) [ 663.068425] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 664.623504] Lustre: Failing over lustre-MDT0000 [ 664.739368] LustreError: 20638:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 664.742918] LustreError: 20638:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 664.789459] Lustre: server umount lustre-MDT0000 complete [ 666.596726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 666.607839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.617980] Lustre: Skipped 1 previous similar message [ 666.629704] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 666.642390] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 684.732570] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 684.735229] LDISKFS-fs (dm-0): recovery complete [ 684.742735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.841055] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 685.048346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 685.082844] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 685.901111] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 688.569283] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 690.155268] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 690.197208] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 690.244766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 690.249214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 695.731706] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 697.223812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 704.817993] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 20:02:28 (1763082148) [ 711.596202] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 713.317484] Lustre: Failing over lustre-MDT0000 [ 713.493872] LustreError: 22463:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 713.501075] LustreError: 22463:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 713.585030] Lustre: server umount lustre-MDT0000 complete [ 715.746614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.749601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 715.765370] Lustre: Skipped 5 previous similar messages [ 732.128436] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082161/real 1763082161] req@ffff94db8a0e3100 x1848724979764992/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763082177 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 732.145829] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 732.737855] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 732.740032] LDISKFS-fs (dm-0): recovery complete [ 732.746367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.375557] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7d71f606bde26155 [ 742.385898] Lustre: MGC192.168.204.105@tcp: Connection restored to 0@lo (at 0@lo) [ 742.388967] Lustre: Skipped 3 previous similar messages [ 742.592751] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 744.337105] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 745.533629] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 748.098566] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 748.146459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 748.146633] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 751.830577] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 753.225608] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 760.744801] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 20:03:24 (1763082204) [ 767.739968] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 769.390430] Lustre: Failing over lustre-MDT0000 [ 769.566596] LustreError: 24288:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 769.574178] LustreError: 24288:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 769.644897] Lustre: server umount lustre-MDT0000 complete [ 789.857552] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 789.859389] LDISKFS-fs (dm-0): recovery complete [ 789.867187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.924812] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 790.146238] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 790.297222] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 793.610260] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 795.174234] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 795.230383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 795.231679] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 800.057649] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 801.362437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 807.941405] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 20:04:12 (1763082252) [ 814.258716] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 815.829511] Lustre: Failing over lustre-MDT0000 [ 815.972441] LustreError: 26113:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 815.976951] LustreError: 26113:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 816.080554] Lustre: server umount lustre-MDT0000 complete [ 816.971122] LustreError: 18219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 817.004874] LustreError: 18219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 75 previous similar messages [ 820.704533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 820.707223] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 820.735676] Lustre: Skipped 8 previous similar messages [ 836.559096] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082265/real 1763082265] req@ffff94db8940ea00 x1848724979827072/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763082281 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 836.632414] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 844.038158] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 844.041604] LDISKFS-fs (dm-0): recovery complete [ 844.071224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.791385] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94db829f5500 x1848724979834112/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 846.096991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.103410] Lustre: Skipped 2 previous similar messages [ 846.149307] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 848.658202] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 849.806283] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 851.442350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 851.445073] Lustre: Skipped 8 previous similar messages [ 851.635292] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 851.671272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 851.676213] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 856.533378] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 857.927212] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 865.332731] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 20:05:09 (1763082309) [ 871.340265] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 872.899223] Lustre: Failing over lustre-MDT0000 [ 873.081100] Lustre: server umount lustre-MDT0000 complete [ 892.976263] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 892.978456] LDISKFS-fs (dm-0): recovery complete [ 892.993802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.156473] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 893.373846] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 896.832325] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 898.681255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 898.681604] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 904.260433] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 905.905639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 913.449830] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 20:05:57 (1763082357) [ 914.154637] Lustre: *** cfs_fail_loc=13b, val=315*** [ 914.156945] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 914.163419] LustreError: 27101:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94db8940ca80 x1848724963862656/t38654705666(0) o35->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:330/0 lens 392/456 e 0 to 0 dl 1763082370 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 920.644606] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 922.250139] Lustre: Failing over lustre-MDT0000 [ 922.384562] LustreError: 29814:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 922.387617] LustreError: 29814:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 922.463837] Lustre: server umount lustre-MDT0000 complete [ 924.129843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 939.984202] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082369/real 1763082369] req@ffff94db88b19f80 x1848724979890944/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763082385 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 942.334359] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 942.336502] LDISKFS-fs (dm-0): recovery complete [ 942.347487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 950.241952] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94db8852e300 x1848724979899776/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 950.511070] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 952.087959] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 952.090927] Lustre: Skipped 1 previous similar message [ 953.729464] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 955.930160] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 955.939944] Lustre: Skipped 1 previous similar message [ 955.948778] Lustre: 27101:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94dc88708700 x1848724963862656/t38654705666(0) o35->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:372/0 lens 392/456 e 0 to 0 dl 1763082412 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 955.992120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 955.992163] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 959.879765] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 961.101996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 967.406015] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 20:06:51 (1763082411) [ 973.455511] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 975.143831] Lustre: Failing over lustre-MDT0000 [ 975.327925] Lustre: server umount lustre-MDT0000 complete [ 976.352035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.356390] Lustre: Skipped 11 previous similar messages [ 992.736102] Lustre: 3653:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082421/real 1763082421] req@ffff94dc88478380 x1848724979922048/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763082437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 992.767974] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 992.779554] LustreError: Skipped 1 previous similar message [ 996.025838] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 996.028114] LDISKFS-fs (dm-0): recovery complete [ 996.034686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.402393] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1007.074578] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1008.612956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1008.616399] Lustre: Skipped 11 previous similar messages [ 1008.726574] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1008.730199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1015.102197] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1016.642738] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1023.664311] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 20:07:47 (1763082467) [ 1029.213504] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1029.948897] Lustre: *** cfs_fail_loc=114, val=0*** [ 1032.393119] Lustre: Failing over lustre-MDT0000 [ 1032.854726] Lustre: server umount lustre-MDT0000 complete [ 1050.064270] Lustre: 3651:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082479/real 1763082479] req@ffff94dca0518700 x1848724979956736/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763082495 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1052.190634] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1052.198136] LDISKFS-fs (dm-0): recovery complete [ 1052.208224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.639328] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1064.791346] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1066.137367] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1066.141662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1072.290419] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1073.795241] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1080.882946] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 20:08:44 (1763082524) [ 1087.887974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1088.844222] Lustre: *** cfs_fail_loc=128, val=0*** [ 1091.336366] Lustre: Failing over lustre-MDT0000 [ 1091.454197] LustreError: 35480:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1091.458129] LustreError: 35480:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 1091.552758] LustreError: 8813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1091.594116] LustreError: 8813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 156 previous similar messages [ 1091.631319] Lustre: server umount lustre-MDT0000 complete [ 1112.892216] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1112.894420] LDISKFS-fs (dm-0): recovery complete [ 1112.908505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.518856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1123.522184] Lustre: Skipped 4 previous similar messages [ 1124.932640] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1124.942212] Lustre: Skipped 2 previous similar messages [ 1127.521000] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1129.028655] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1129.042747] Lustre: Skipped 2 previous similar messages [ 1129.075782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1129.079074] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1135.444257] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1136.950696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1145.384622] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 20:09:49 (1763082589) [ 1152.117994] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1154.351351] Lustre: Failing over lustre-MDT0000 [ 1154.527946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1154.532169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.685478] Lustre: server umount lustre-MDT0000 complete [ 1174.833393] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082604/real 1763082604] req@ffff94dcad962d80 x1848724980025344/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763082620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1174.855243] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1174.867531] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1174.886434] LustreError: Skipped 2 previous similar messages [ 1178.654323] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1178.660929] LDISKFS-fs (dm-0): recovery complete [ 1178.670266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.250879] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7d71f606bde28afa [ 1185.480782] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1185.489597] Lustre: Skipped 1 previous similar message [ 1189.520331] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1191.320326] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1191.328128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1197.352924] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1199.171981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1207.377561] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 20:10:51 (1763082651) [ 1216.680559] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1219.171611] Lustre: Failing over lustre-MDT0000 [ 1219.386674] Lustre: server umount lustre-MDT0000 complete [ 1241.835722] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1241.839419] LDISKFS-fs (dm-0): recovery complete [ 1241.849717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1248.224643] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94db88928380 x1848724980069120/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1252.732681] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1254.067159] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1254.072071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1260.432958] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1262.157544] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1270.806896] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 20:11:54 (1763082714) [ 1278.134916] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1285.604835] Lustre: Failing over lustre-MDT0000 [ 1285.800792] Lustre: server umount lustre-MDT0000 complete [ 1289.696241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.709317] Lustre: Skipped 19 previous similar messages [ 1305.559767] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763082734/real 1763082734] req@ffff94db880dbb80 x1848724980100352/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763082750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1305.608406] Lustre: 3652:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1307.263531] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1307.265637] LDISKFS-fs (dm-0): recovery complete [ 1307.281485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.786477] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94db8a20ed80 x1848724980107904/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1314.964064] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1314.970619] Lustre: Skipped 1 previous similar message [ 1318.773492] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1320.422167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1320.437835] Lustre: Skipped 20 previous similar messages [ 1323.694638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1323.712308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1327.347384] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1328.924801] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1351.646901] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 20:13:15 (1763082795) [ 1358.384848] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1360.141590] Lustre: Failing over lustre-MDT0000 [ 1360.255203] LustreError: 42837:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1360.262196] LustreError: 42837:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 1360.360378] Lustre: server umount lustre-MDT0000 complete [ 1380.950222] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1380.952357] LDISKFS-fs (dm-0): recovery complete [ 1380.960541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1386.979765] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94dcbc2cd180 x1848724980204928/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1388.816599] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1388.819288] Lustre: Skipped 3 previous similar messages [ 1390.605216] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1392.648126] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1392.650720] Lustre: Skipped 3 previous similar messages [ 1392.675171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1392.675386] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1397.531437] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1398.872886] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1409.468541] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 20:14:13 (1763082853) [ 1417.468930] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1419.676779] Lustre: Failing over lustre-MDT0000 [ 1419.959681] Lustre: server umount lustre-MDT0000 complete [ 1423.330818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1439.712145] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1439.719386] LustreError: Skipped 3 previous similar messages [ 1443.007530] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1443.009382] LDISKFS-fs (dm-0): recovery complete [ 1443.020440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.954414] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94dcb79f3b80 x1848724980253056/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1454.911968] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1455.675612] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1455.680419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1462.114475] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1463.753769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1472.272533] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 20:15:16 (1763082916) [ 1479.538589] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1481.421843] Lustre: Failing over lustre-MDT0000 [ 1481.690172] Lustre: server umount lustre-MDT0000 complete [ 1503.832130] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1503.834410] LDISKFS-fs (dm-0): recovery complete [ 1503.854344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1517.021975] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1518.229806] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1518.234027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1525.001444] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1526.761830] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1534.251670] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 20:16:18 (1763082978) [ 1541.776536] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1543.780542] Lustre: Failing over lustre-MDT0000 [ 1544.059224] Lustre: server umount lustre-MDT0000 complete [ 1564.333076] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1564.335140] LDISKFS-fs (dm-0): recovery complete [ 1564.350276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.264028] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1569.911942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1569.912308] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1573.672586] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1574.803600] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1580.630343] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 20:17:04 (1763083024) [ 1586.382640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1587.938169] Lustre: Failing over lustre-MDT0000 [ 1588.258718] Lustre: server umount lustre-MDT0000 complete [ 1603.552361] LustreError: 7806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1603.574186] LustreError: 7806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 268 previous similar messages [ 1604.575937] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763083033/real 1763083033] req@ffff94dcad953480 x1848724980340864/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763083049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1604.611957] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1608.825454] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1608.827539] LDISKFS-fs (dm-0): recovery complete [ 1608.833291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1614.031508] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1614.034044] Lustre: Skipped 4 previous similar messages [ 1616.929040] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1619.520420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1619.522685] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1623.330723] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1624.919623] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1632.217705] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 20:17:56 (1763083076) [ 1639.095804] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1641.261968] Lustre: Failing over lustre-MDT0000 [ 1641.508664] Lustre: server umount lustre-MDT0000 complete [ 1661.508588] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1661.510728] LDISKFS-fs (dm-0): recovery complete [ 1661.517916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.655220] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7d71f606bde34b57 [ 1671.904719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1671.914779] Lustre: Skipped 8 previous similar messages [ 1675.308677] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1677.391024] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1677.391685] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1681.949191] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1683.399317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1691.065478] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 20:18:55 (1763083135) [ 1697.358485] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1699.140312] Lustre: Failing over lustre-MDT0000 [ 1699.393454] Lustre: server umount lustre-MDT0000 complete [ 1702.884937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1702.889372] LustreError: Skipped 1 previous similar message [ 1719.022751] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1719.028358] LDISKFS-fs (dm-0): recovery complete [ 1719.035767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1722.304408] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1724.517375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1724.519258] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1728.095592] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1729.506126] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1736.645939] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 20:19:40 (1763083180) [ 1742.345439] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1744.078454] Lustre: Failing over lustre-MDT0000 [ 1746.311633] Lustre: server umount lustre-MDT0000 complete [ 1768.477882] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1768.485355] LDISKFS-fs (dm-0): recovery complete [ 1768.500371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.611460] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7d71f606bde356f6 [ 1780.386687] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1782.322056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1782.326795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1787.137677] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1788.850296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1796.103972] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 20:20:40 (1763083240) [ 1802.582540] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1804.167491] Lustre: Failing over lustre-MDT0000 [ 1804.339264] Lustre: server umount lustre-MDT0000 complete [ 1807.840053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.851128] Lustre: Skipped 32 previous similar messages [ 1824.287388] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1824.289040] LDISKFS-fs (dm-0): recovery complete [ 1824.302884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.490477] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1829.926801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 1829.930019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 1834.369380] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1836.032849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1842.882306] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 20:21:27 (1763083287) [ 1848.291415] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1849.710988] Lustre: Failing over lustre-MDT0000 [ 1849.955161] Lustre: server umount lustre-MDT0000 complete [ 1870.177648] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1870.180737] LDISKFS-fs (dm-0): recovery complete [ 1870.193901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.363158] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1882.089179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1882.097735] Lustre: Skipped 41 previous similar messages [ 1882.247882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 1882.251858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:897) [ 1887.489342] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1889.117186] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1896.587265] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 20:22:20 (1763083340) [ 1902.946786] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1904.608634] Lustre: Failing over lustre-MDT0000 [ 1904.728386] LustreError: 61086:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1904.734425] LustreError: 61086:0:(obd_class.h:479:obd_check_dev()) Skipped 79 previous similar messages [ 1904.813079] Lustre: server umount lustre-MDT0000 complete [ 1924.622055] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1924.624623] LDISKFS-fs (dm-0): recovery complete [ 1924.640141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.219553] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1935.222482] Lustre: Skipped 9 previous similar messages [ 1936.436075] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1939.026629] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1939.032867] Lustre: Skipped 9 previous similar messages [ 1939.072094] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1939.073188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1942.677263] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1944.104222] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1950.748687] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 20:23:14 (1763083394) [ 1956.307503] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1957.945301] Lustre: Failing over lustre-MDT0000 [ 1958.125494] Lustre: server umount lustre-MDT0000 complete [ 1974.751177] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1974.756350] LustreError: Skipped 9 previous similar messages [ 1976.385986] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1976.388475] LDISKFS-fs (dm-0): recovery complete [ 1976.401842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1984.992166] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94dcbfaff100 x1848724980569216/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1985.013084] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 1988.088732] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1990.770468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 1990.773938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 1994.645146] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1995.842795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2001.686383] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 20:24:05 (1763083445) [ 2006.867268] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2008.552645] Lustre: Failing over lustre-MDT0000 [ 2008.728685] Lustre: server umount lustre-MDT0000 complete [ 2011.106298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.111505] LustreError: Skipped 4 previous similar messages [ 2028.761626] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2028.763768] LDISKFS-fs (dm-0): recovery complete [ 2028.771447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.224692] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7d71f606bde371c2 [ 2040.396364] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2042.942607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:993) [ 2042.945779] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 2046.378309] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2047.739086] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2053.960126] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 20:24:58 (1763083498) [ 2059.220122] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2060.738229] Lustre: Failing over lustre-MDT0000 [ 2060.896743] Lustre: server umount lustre-MDT0000 complete [ 2080.094254] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2080.096788] LDISKFS-fs (dm-0): recovery complete [ 2080.103263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.932352] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2094.671069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2094.671214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2097.337778] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2098.597810] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2105.868253] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 20:25:49 (1763083549) [ 2111.939232] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2113.439974] Lustre: Failing over lustre-MDT0000 [ 2113.596533] Lustre: server umount lustre-MDT0000 complete [ 2131.408106] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763083560/real 1763083560] req@ffff94dcb77b6a00 x1848724980655104/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763083576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2131.419656] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 2132.569649] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2132.570970] LDISKFS-fs (dm-0): recovery complete [ 2132.587466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.668317] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7d71f606bde37e1e [ 2141.938489] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2141.944051] Lustre: Skipped 9 previous similar messages [ 2145.551479] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2147.387365] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2147.387446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2150.910745] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2152.054777] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2157.659769] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:26:41 (1763083601) [ 2162.366364] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2163.463962] Lustre: Failing over lustre-MDT0000 [ 2163.609328] Lustre: server umount lustre-MDT0000 complete [ 2181.541527] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2181.543419] LDISKFS-fs (dm-0): recovery complete [ 2181.548793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.613171] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2249.716287] Lustre: 70862:0:(ldlm_lib.c:2929:target_recovery_thread()) too long recovery - read logs [ 2249.722112] LustreError: dumping log to /tmp/lustre-log.1763083695.70862 [ 2249.865085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2249.869177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2252.915413] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2254.109933] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2260.527822] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 20:28:24 (1763083704) [ 2264.384452] Lustre: 71717:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b3c485f8-b8b5-445b-b430-6dd849e35313 at adminstrative request [ 2269.474292] Lustre: Failing over lustre-MDT0000 [ 2269.662363] Lustre: server umount lustre-MDT0000 complete [ 2270.177215] LustreError: 7806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2270.186316] LustreError: 7806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 332 previous similar messages [ 2286.507197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.811720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.814464] Lustre: Skipped 10 previous similar messages [ 2290.005497] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2292.311956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2292.312827] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2296.400366] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2298.037492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2302.110542] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2308.406070] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2313.173462] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 20:29:17 (1763083757) [ 2313.907211] Lustre: 73568:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b3c485f8-b8b5-445b-b430-6dd849e35313 at adminstrative request [ 2321.854430] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:29:25 (1763083765) [ 2327.735965] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2329.370855] Lustre: Failing over lustre-MDT0000 [ 2329.536213] Lustre: server umount lustre-MDT0000 complete [ 2349.164142] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2349.166364] LDISKFS-fs (dm-0): recovery complete [ 2349.175679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.322689] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2354.781090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2354.782828] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2358.475072] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2359.564282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2365.938259] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:30:10 (1763083810) [ 2371.894817] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2373.527577] Lustre: Failing over lustre-MDT0000 [ 2373.706534] Lustre: server umount lustre-MDT0000 complete [ 2393.528316] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2393.531257] LDISKFS-fs (dm-0): recovery complete [ 2393.538508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.945898] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2406.423547] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2406.424130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2409.652435] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2411.001714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2417.929411] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:31:01 (1763083861) [ 2423.467787] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2424.992041] Lustre: Failing over lustre-MDT0000 [ 2425.150716] Lustre: server umount lustre-MDT0000 complete [ 2426.848784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2426.855026] Lustre: Skipped 46 previous similar messages [ 2444.445543] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2444.447915] LDISKFS-fs (dm-0): recovery complete [ 2444.461522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.475461] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94dcbe674e00 x1848724980842752/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2453.485496] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 2456.476305] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2459.172218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2459.178785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2462.267870] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2463.864937] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2471.612289] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 20:31:55 (1763083915) [ 2477.133301] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2478.494915] Lustre: Failing over lustre-MDT0000 [ 2478.698482] Lustre: server umount lustre-MDT0000 complete [ 2497.283385] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2497.285450] LDISKFS-fs (dm-0): recovery complete [ 2497.291203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.211977] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7d71f606bde39e62 [ 2506.220437] Lustre: MGC192.168.204.105@tcp: Connection restored to 0@lo (at 0@lo) [ 2506.224900] Lustre: Skipped 45 previous similar messages [ 2508.632156] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2511.398126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2511.409973] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2514.546207] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2516.008508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2521.947189] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:32:46 (1763083966) [ 2527.217804] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2528.607417] Lustre: Failing over lustre-MDT0000 [ 2528.733164] LustreError: 81555:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2528.735565] LustreError: 81555:0:(obd_class.h:479:obd_check_dev()) Skipped 87 previous similar messages [ 2528.767450] Lustre: server umount lustre-MDT0000 complete [ 2531.809450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2531.817063] LustreError: Skipped 8 previous similar messages [ 2545.682454] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2545.684587] LDISKFS-fs (dm-0): recovery complete [ 2545.691814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2547.472414] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2547.485492] Lustre: Skipped 10 previous similar messages [ 2547.867712] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2551.291412] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2551.298845] Lustre: Skipped 10 previous similar messages [ 2551.327940] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 2551.328636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 2553.713057] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2554.876119] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2560.913444] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:33:25 (1763084005) [ 2565.725079] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2567.183645] Lustre: Failing over lustre-MDT0000 [ 2567.355598] Lustre: server umount lustre-MDT0000 complete [ 2584.524858] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2584.526870] LDISKFS-fs (dm-0): recovery complete [ 2584.536790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.600318] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2584.606195] LustreError: Skipped 10 previous similar messages [ 2586.873519] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2590.255839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2590.255948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2592.391789] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2593.307861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2599.074765] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:34:03 (1763084043) [ 2602.757149] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2603.929748] Lustre: Failing over lustre-MDT0000 [ 2604.095738] Lustre: server umount lustre-MDT0000 complete [ 2621.020043] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2621.022176] LDISKFS-fs (dm-0): recovery complete [ 2621.027292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.646149] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2626.604799] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 2626.605225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 2629.067492] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2630.196112] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2635.501924] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:34:39 (1763084079) [ 2640.062485] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2641.302200] Lustre: Failing over lustre-MDT0000 [ 2641.470386] Lustre: server umount lustre-MDT0000 complete [ 2659.955668] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2659.959935] LDISKFS-fs (dm-0): recovery complete [ 2659.970128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.841916] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2665.514207] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 2665.514225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 2668.212086] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2669.364883] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2675.395616] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:35:19 (1763084119) [ 2680.127145] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2681.373480] Lustre: Failing over lustre-MDT0000 [ 2681.590099] Lustre: server umount lustre-MDT0000 complete [ 2699.245285] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2699.247250] LDISKFS-fs (dm-0): recovery complete [ 2699.253870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.800295] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2704.929591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 2704.933350] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 2707.172938] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2708.504883] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2714.068295] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:35:58 (1763084158) [ 2717.996618] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2718.972990] Lustre: Failing over lustre-MDT0000 [ 2719.158839] Lustre: server umount lustre-MDT0000 complete [ 2735.936955] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2735.939066] LDISKFS-fs (dm-0): recovery complete [ 2735.944812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.245063] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2741.264074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 2741.264074] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 2742.954828] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2743.725590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2748.659768] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 20:36:33 (1763084193) [ 2753.557970] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2754.673273] Lustre: Failing over lustre-MDT0000 [ 2754.826019] Lustre: server umount lustre-MDT0000 complete [ 2771.493722] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2771.499518] LDISKFS-fs (dm-0): recovery complete [ 2771.509523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.761225] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2771.766720] Lustre: Skipped 12 previous similar messages [ 2773.563467] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2777.114152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 2777.117525] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 2779.240281] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2780.157239] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2785.149045] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 20:37:09 (1763084229) [ 2785.686216] Lustre: 93875:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b3c485f8-b8b5-445b-b430-6dd849e35313 at adminstrative request [ 2792.333984] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 20:37:16 (1763084236) [ 2796.295640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2797.363287] Lustre: Failing over lustre-MDT0000 [ 2797.528866] Lustre: server umount lustre-MDT0000 complete [ 2804.531578] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2804.533948] LDISKFS-fs (dm-0): recovery complete [ 2804.543123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2804.720363] Lustre: lustre-MDT0000: Aborting client recovery [ 2804.722337] LustreError: 95155:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2804.725688] Lustre: 95188:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2804.729316] Lustre: 95188:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client b3c485f8-b8b5-445b-b430-6dd849e35313@ [ 2804.734343] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2804.739688] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2804.746978] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2804.780755] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 2804.780755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505) [ 2806.491246] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2809.824421] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2820.019038] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 20:37:44 (1763084264) [ 2823.785814] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2824.823019] Lustre: Failing over lustre-MDT0000 [ 2824.987153] Lustre: server umount lustre-MDT0000 complete [ 2832.047576] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2832.049517] LDISKFS-fs (dm-0): recovery complete [ 2832.057917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.286148] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2832.304838] Lustre: lustre-MDT0000: Aborting client recovery [ 2832.307396] LustreError: 97001:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2832.311963] Lustre: 97034:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2832.317533] Lustre: 97034:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 2832.322860] Lustre: 97034:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2832.331300] Lustre: 97034:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2832.338156] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2832.342836] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 2832.353096] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 2832.394995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1537) [ 2832.405477] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1537) [ 2834.426118] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2837.479159] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2845.277128] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2845.281768] Lustre: Skipped 1 previous similar message [ 2849.351625] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 20:38:13 (1763084293) [ 2854.367162] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2855.465579] Lustre: Failing over lustre-MDT0000 [ 2855.656984] Lustre: server umount lustre-MDT0000 complete [ 2862.291063] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2862.293319] LDISKFS-fs (dm-0): recovery complete [ 2862.306190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.546632] Lustre: lustre-MDT0000: Aborting client recovery [ 2862.552196] LustreError: 98849:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2862.561764] Lustre: 98882:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2862.571357] Lustre: 98882:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 2862.579768] Lustre: 98882:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2862.591546] Lustre: 98882:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2862.602370] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2862.624076] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 2862.644598] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 2862.691610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 2862.700290] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 2864.819881] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2867.681119] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2879.187379] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 20:38:43 (1763084323) [ 2879.813913] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2879.828908] LustreError: 6517:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94db88847800 x1848724964893824/t201863462916(0) o36->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:31/0 lens 512/456 e 0 to 0 dl 1763084336 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 2882.828777] Lustre: Failing over lustre-MDT0000 [ 2882.967209] Lustre: server umount lustre-MDT0000 complete [ 2883.043307] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.051424] LustreError: 6522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 288 previous similar messages [ 2887.578570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2887.789416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.792634] Lustre: Skipped 14 previous similar messages [ 2887.817263] Lustre: lustre-MDT0000: Aborting client recovery [ 2887.819711] LustreError: 100156:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2887.819818] Lustre: 100189:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2887.827540] Lustre: 100189:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 2887.831185] Lustre: 100189:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2887.836283] Lustre: 100189:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2887.840147] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2887.844377] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 2887.853415] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 2887.878389] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 2887.878546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1601) [ 2889.823546] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2893.280750] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2904.038557] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 2905.040874] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 20:39:09 (1763084349) [ 2909.688256] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2911.420707] Lustre: Failing over lustre-MDT0000 [ 2911.630472] Lustre: server umount lustre-MDT0000 complete [ 2918.646643] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2918.648755] LDISKFS-fs (dm-0): recovery complete [ 2918.655524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.865132] Lustre: lustre-MDT0000: Aborting client recovery [ 2918.867533] LustreError: 102101:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2918.871706] Lustre: 102134:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2918.874979] Lustre: 102134:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 2918.879335] Lustre: 102134:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2918.884757] Lustre: 102134:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2918.888674] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2918.894403] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 2918.907172] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 2918.940774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1633) [ 2918.946392] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 2921.037764] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2924.001199] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2935.037358] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:39:39 (1763084379) [ 2958.137772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2959.101652] Lustre: Failing over lustre-MDT0000 [ 2959.143162] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 2959.146463] Lustre: Skipped 3 previous similar messages [ 2959.240166] Lustre: server umount lustre-MDT0000 complete [ 2975.650888] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2975.653146] LDISKFS-fs (dm-0): recovery complete [ 2975.660707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.125725] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2981.419361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 2981.419415] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 2983.247912] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2984.066954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2995.913704] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 20:40:40 (1763084440) [ 3010.677396] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3016.113227] Lustre: Failing over lustre-MDT0000 [ 3016.197996] LustreError: 3651:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff94dcada2d500 x1848724981386240/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3016.416496] Lustre: server umount lustre-MDT0000 complete [ 3031.612480] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3031.614525] LDISKFS-fs (dm-0): recovery complete [ 3031.619978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.408034] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3038.516055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3038.518542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3040.120970] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3040.832895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3054.291335] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 20:41:38 (1763084498) [ 3055.290910] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3055.756714] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.761752] Lustre: Skipped 60 previous similar messages [ 3055.764018] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3055.767233] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3059.062141] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 20:41:43 (1763084503) [ 3072.290661] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3077.145362] Lustre: Failing over lustre-OST0000 [ 3077.208705] Lustre: server umount lustre-OST0000 complete [ 3093.058292] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3093.060031] LDISKFS-fs (dm-2): recovery complete [ 3093.065115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3095.934351] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3142.417309] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 20:43:07 (1763084587) [ 3145.426376] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3146.462490] Lustre: Failing over lustre-MDT0000 [ 3146.588324] LustreError: 110591:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3146.590708] LustreError: 110591:0:(obd_class.h:479:obd_check_dev()) Skipped 115 previous similar messages [ 3146.619178] Lustre: server umount lustre-MDT0000 complete [ 3161.359613] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3161.361574] LDISKFS-fs (dm-0): recovery complete [ 3161.365911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3162.902220] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3163.918570] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3163.922265] Lustre: Skipped 9 previous similar messages [ 3166.689234] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3166.692483] Lustre: Skipped 63 previous similar messages [ 3166.702475] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3166.706725] Lustre: Skipped 9 previous similar messages [ 3166.722316] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3166.722360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3168.131498] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3168.808342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3183.024605] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 20:43:47 (1763084627) [ 3183.071197] Lustre: 111230:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763084612/real 1763084612] req@ffff94dcbe7cea00 x1848724981707648/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1763084628 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 3183.082627] Lustre: 111230:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3183.086207] LustreError: 111230:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3183.087597] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3184.095487] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3185.085913] LustreError: 6518:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3190.239129] LustreError: 6518:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3190.242610] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnecting [ 3190.923710] LustreError: 6518:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3196.383323] LustreError: 6518:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3196.387554] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnecting [ 3197.058367] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3202.527136] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3202.530772] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnecting [ 3203.267669] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3208.672110] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3209.385905] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3214.815238] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3214.818587] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnecting [ 3214.822121] Lustre: Skipped 1 previous similar message [ 3221.673408] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3221.677422] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3227.103131] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3227.106526] LustreError: 6517:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 3233.247201] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnecting [ 3233.251298] Lustre: Skipped 2 previous similar messages [ 3240.168911] LustreError: 8813:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 3240.172297] LustreError: 8813:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3245.535134] LustreError: 8813:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3245.538422] LustreError: 8813:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 3248.791948] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 20:44:53 (1763084693) [ 3249.532804] LustreError: 8813:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3259.663234] Lustre: lustre-MDT0000: Export ffff94dcbc39a800 already connecting from 192.168.204.5@tcp [ 3264.782761] Lustre: lustre-MDT0000: Export ffff94dcbc39a800 already connecting from 192.168.204.5@tcp [ 3269.905690] Lustre: lustre-MDT0000: Export ffff94dcbc39a800 already connecting from 192.168.204.5@tcp [ 3275.023674] Lustre: lustre-MDT0000: Export ffff94dcbc39a800 already connecting from 192.168.204.5@tcp [ 3275.028391] Lustre: Skipped 1 previous similar message [ 3280.143263] Lustre: lustre-MDT0000: Export ffff94dcbc39a800 already connecting from 192.168.204.5@tcp [ 3289.591219] LustreError: 8813:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3289.594979] Lustre: 8813:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94dcb77ba680 x1848724967532800/t0(0) o38->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:0/0 lens 520/416 e 0 to 0 dl 1763084714 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3290.382774] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnecting [ 3290.386641] Lustre: Skipped 3 previous similar messages [ 3290.388802] LustreError: 6518:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3314.959280] Lustre: lustre-MDT0000: Export ffff94dcbc39a800 already connecting from 192.168.204.5@tcp [ 3314.962713] Lustre: Skipped 1 previous similar message [ 3330.431188] LustreError: 6518:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3330.435239] Lustre: 6518:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94dcb77b8e00 x1848724967537536/t0(0) o38->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:0/0 lens 520/416 e 0 to 0 dl 1763084755 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3334.960267] LustreError: 18219:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3360.527495] Lustre: lustre-MDT0000: Export ffff94dcbc39a800 already connecting from 192.168.204.5@tcp [ 3360.530481] Lustre: Skipped 3 previous similar messages [ 3375.007328] LustreError: 18219:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3375.011342] Lustre: 18219:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94dcbfc89c00 x1848724967541120/t0(0) o38->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:0/0 lens 520/416 e 0 to 0 dl 1763084800 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3375.890513] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnecting [ 3375.895431] Lustre: Skipped 1 previous similar message [ 3375.899089] LustreError: 8813:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3401.487346] Lustre: lustre-MDT0000: Export ffff94dcbc39a800 already connecting from 192.168.204.5@tcp [ 3401.491417] Lustre: Skipped 2 previous similar messages [ 3415.943128] LustreError: 8813:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3415.946796] Lustre: 8813:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94dc90375f80 x1848724967544064/t0(0) o38->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:0/0 lens 520/416 e 0 to 0 dl 1763084841 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3416.847041] LustreError: 6518:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3456.895299] LustreError: 6518:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3456.898878] Lustre: 6518:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94dcbe7cc380 x1848724967547136/t0(0) o38->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:0/0 lens 520/416 e 0 to 0 dl 1763084882 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3457.806791] LustreError: 6518:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3462.279121] LustreError: 6518:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 3464.216518] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 20:48:28 (1763084908) [ 3467.553509] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3469.414838] Lustre: Failing over lustre-MDT0000 [ 3469.525834] Lustre: server umount lustre-MDT0000 complete [ 3474.653081] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3474.654876] LDISKFS-fs (dm-0): recovery complete [ 3474.659852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.714797] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3474.719042] LustreError: Skipped 13 previous similar messages [ 3474.789950] Lustre: *** cfs_fail_loc=712, val=0*** [ 3474.792612] LustreError: 42136:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff94db829f4e00 x1848724981856512/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 3474.841937] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3474.842085] Lustre: lustre-MDT0000: Aborting client recovery [ 3474.846547] Lustre: Skipped 19 previous similar messages [ 3474.848796] LustreError: 115960:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3474.851357] Lustre: 115993:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3474.864391] Lustre: 115993:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 3474.870629] Lustre: 115993:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3474.875087] Lustre: 115993:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3474.879509] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3474.883436] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 3474.893883] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3474.925319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 3474.925464] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 3476.550743] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3480.034916] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3487.020187] Lustre: Failing over lustre-MDT0000 [ 3487.203215] Lustre: server umount lustre-MDT0000 complete [ 3488.529130] LustreError: 18219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3488.536395] LustreError: 18219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 70 previous similar messages [ 3490.271681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3490.275151] LustreError: Skipped 11 previous similar messages [ 3500.600050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3500.719454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.722542] Lustre: Skipped 6 previous similar messages [ 3502.175520] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3506.165475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 3506.165505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 3507.439631] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3508.054642] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3511.644064] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 20:49:16 (1763084956) [ 3511.683641] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnecting [ 3511.687099] Lustre: Skipped 2 previous similar messages [ 3514.953894] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 20:49:19 (1763084959) [ 3515.286558] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3515.288629] LustreError: 117719:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dcb76bc000 x1848724967604608/t0(0) o700->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:666/0 lens 264/248 e 0 to 0 dl 1763084971 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 3532.758820] Lustre: Failing over lustre-MDT0000 [ 3532.958391] Lustre: server umount lustre-MDT0000 complete [ 3546.234068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3547.859017] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3551.737513] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 3551.737513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 3553.107751] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3553.769061] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3558.433535] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 20:50:03 (1763085003) [ 3559.409909] Lustre: Failing over lustre-OST0000 [ 3559.475891] Lustre: server umount lustre-OST0000 complete [ 3572.473510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3574.543041] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3577.837785] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3578.454837] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3644.117293] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 20:51:28 (1763085088) [ 3646.973354] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3647.822786] Lustre: Failing over lustre-MDT0000 [ 3647.982089] Lustre: server umount lustre-MDT0000 complete [ 3662.450253] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3662.451609] LDISKFS-fs (dm-0): recovery complete [ 3662.455644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3663.958723] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3668.042511] Lustre: *** cfs_fail_loc=216, val=0*** [ 3668.042521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 3668.044394] LustreError: 121769:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 3669.087487] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 3730.974435] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 20:52:55 (1763085175) [ 3731.656559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3731.662287] Lustre: Skipped 25 previous similar messages [ 3739.356723] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 20:53:03 (1763085183) [ 3740.488170] Lustre: Failing over lustre-MDT0000 [ 3740.683073] Lustre: server umount lustre-MDT0000 complete [ 3754.212619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.937481] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3759.593037] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 3759.595278] LustreError: 123333:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dcb7510e00 x1848724967724288/t0(0) o101->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:195/0 lens 328/344 e 0 to 0 dl 1763085255 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 3814.158581] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [ 3814.166485] Lustre: 123333:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3814.175943] Lustre: lustre-MDT0000: Recovery over after 1:00, of 2 clients 2 recovered and 0 were evicted. [ 3814.175975] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3814.179752] Lustre: Skipped 4 previous similar messages [ 3814.182340] Lustre: Skipped 27 previous similar messages [ 3814.200671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 3814.200713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 3815.709883] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3816.327248] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3820.374076] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 20:54:25 (1763085265) [ 3821.831113] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3825.007340] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3825.784117] Lustre: Failing over lustre-MDT0000 [ 3825.921640] LustreError: 124676:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3825.924479] LustreError: 124676:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 3825.948312] Lustre: server umount lustre-MDT0000 complete [ 3840.530872] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3840.532875] LDISKFS-fs (dm-0): recovery complete [ 3840.537207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.806857] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3841.810684] Lustre: Skipped 5 previous similar messages [ 3842.123934] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3846.135806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3137) [ 3846.136138] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 3847.682359] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3848.396735] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3853.256160] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 20:54:57 (1763085297) [ 3853.672316] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3857.884465] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3858.617931] Lustre: Failing over lustre-MDT0000 [ 3858.720878] Lustre: server umount lustre-MDT0000 complete [ 3873.593992] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3873.595873] LDISKFS-fs (dm-0): recovery complete [ 3873.599402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3875.330254] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3878.902870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3169) [ 3878.902988] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 3880.221177] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3880.867551] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3884.888743] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 20:55:29 (1763085329) [ 3885.288948] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3889.664944] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3890.490560] Lustre: Failing over lustre-MDT0000 [ 3890.616468] Lustre: server umount lustre-MDT0000 complete [ 3906.010892] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3906.012542] LDISKFS-fs (dm-0): recovery complete [ 3906.017493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.855068] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3911.671053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 3911.671082] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3169) [ 3916.678950] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 20:56:01 (1763085361) [ 3918.132400] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3918.134554] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 3918.136894] LustreError: 27101:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dcbfe8f800 x1848724967776000/t257698037777(0) o35->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:314/0 lens 392/456 e 0 to 0 dl 1763085374 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3919.353166] Lustre: Failing over lustre-MDT0000 [ 3919.539821] Lustre: server umount lustre-MDT0000 complete [ 3933.107051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.875443] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3938.293477] Lustre: 27101:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94dcb7ba5500 x1848724967776000/t257698037777(0) o35->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:334/0 lens 392/456 e 0 to 0 dl 1763085394 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3938.297870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3201) [ 3938.298415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 3939.762656] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3940.403908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3944.528373] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 20:56:29 (1763085389) [ 3945.038691] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3945.040840] LustreError: 6517:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dcada2c000 x1848724967789696/t261993005072(0) o36->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:385/0 lens 504/448 e 0 to 0 dl 1763085445 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3949.196295] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3950.035572] Lustre: Failing over lustre-MDT0000 [ 3950.142393] Lustre: server umount lustre-MDT0000 complete [ 3965.363468] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3965.365411] LDISKFS-fs (dm-0): recovery complete [ 3965.369599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3967.193689] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3971.065557] Lustre: 112493:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94dcb2e71f80 x1848724967789696/t261993005072(0) o36->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:411/0 lens 504/2880 e 0 to 0 dl 1763085471 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3971.067248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 3971.068280] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3233) [ 3972.658774] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3973.388388] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3977.528376] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 20:57:02 (1763085422) [ 3977.999536] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3978.001786] LustreError: 18219:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dcb77bb100 x1848724967803392/t266287972368(0) o36->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:418/0 lens 504/448 e 0 to 0 dl 1763085478 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3979.360626] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3982.230843] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3982.939901] Lustre: Failing over lustre-MDT0000 [ 3983.058142] Lustre: server umount lustre-MDT0000 complete [ 3998.113338] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3998.115108] LDISKFS-fs (dm-0): recovery complete [ 3998.120228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.888631] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4003.313498] Lustre: 27101:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94dcb760ad80 x1848724967803648/t266287972369(0) o35->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:399/0 lens 392/456 e 0 to 0 dl 1763085459 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4003.319705] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3265) [ 4003.319706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 4003.322716] Lustre: 27101:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4008.222168] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 20:57:32 (1763085452) [ 4008.616710] LustreError: 18211:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dcbc2c1880 x1848724967816320/t270582939664(0) o36->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:448/0 lens 504/448 e 0 to 0 dl 1763085508 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4008.628217] LustreError: 18211:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 4009.938246] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4009.940480] Lustre: Skipped 1 previous similar message [ 4012.992573] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4013.706895] Lustre: Failing over lustre-MDT0000 [ 4013.806786] Lustre: server umount lustre-MDT0000 complete [ 4028.686523] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4028.688400] LDISKFS-fs (dm-0): recovery complete [ 4028.693669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.412618] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4034.042567] Lustre: 6517:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94dcb7ba6300 x1848724967816320/t270582939664(0) o36->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:474/0 lens 504/2880 e 0 to 0 dl 1763085534 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4034.043983] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 4034.046436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3329) [ 4038.629948] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 20:58:03 (1763085483) [ 4039.036692] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4040.401198] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4040.403068] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4040.405490] Lustre: Skipped 2 previous similar messages [ 4040.407845] LustreError: 27101:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dc884fce00 x1848724967828864/t274877906960(0) o35->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:436/0 lens 392/456 e 0 to 0 dl 1763085496 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4044.285405] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4045.029909] Lustre: Failing over lustre-MDT0000 [ 4045.138068] Lustre: server umount lustre-MDT0000 complete [ 4060.471132] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4060.473360] LDISKFS-fs (dm-0): recovery complete [ 4060.479647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4062.167178] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4065.779139] Lustre: 27101:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94dcbb899180 x1848724967828864/t274877906960(0) o35->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:462/0 lens 392/456 e 0 to 0 dl 1763085522 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4065.782410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 4065.782546] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 4070.736299] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 20:58:35 (1763085515) [ 4071.136702] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4131.610172] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnecting [ 4131.613858] Lustre: Skipped 4 previous similar messages [ 4131.616558] Lustre: 112493:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94db89be7100 x1848724967839360/t279172874255(0) o101->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:571/0 lens 664/3488 e 0 to 0 dl 1763085631 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4133.893112] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 20:59:38 (1763085578) [ 4136.475784] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4137.153851] Lustre: Failing over lustre-MDT0000 [ 4137.238517] Lustre: server umount lustre-MDT0000 complete [ 4137.439758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4137.439894] LustreError: 18219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.441967] LustreError: Skipped 12 previous similar messages [ 4137.448876] LustreError: 18219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 178 previous similar messages [ 4151.432960] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4151.434404] LDISKFS-fs (dm-0): recovery complete [ 4151.439159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.479965] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4151.484416] LustreError: Skipped 12 previous similar messages [ 4151.550864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4151.553866] Lustre: Skipped 12 previous similar messages [ 4151.573526] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4151.576047] Lustre: Skipped 15 previous similar messages [ 4152.860145] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4156.923621] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3361) [ 4156.923653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 4158.091162] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4158.642815] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4172.036372] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 21:00:16 (1763085616) [ 4174.893890] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4175.539161] Lustre: Failing over lustre-MDT0000 [ 4175.729485] Lustre: server umount lustre-MDT0000 complete [ 4189.946405] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4189.948145] LDISKFS-fs (dm-0): recovery complete [ 4189.952142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.438576] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4195.320940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3425) [ 4195.320958] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 4196.475551] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4197.022675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4199.368733] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4204.528088] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 21:00:49 (1763085649) [ 4214.083110] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4214.718951] Lustre: Failing over lustre-MDT0000 [ 4214.973678] Lustre: server umount lustre-MDT0000 complete [ 4228.774064] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4228.776150] LDISKFS-fs (dm-0): recovery complete [ 4228.779393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4230.106129] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4234.287066] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 4234.287098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 4235.461646] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4235.997271] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4249.595248] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 21:01:34 (1763085694) [ 4253.262364] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4253.908097] Lustre: Failing over lustre-MDT0000 [ 4254.029747] Lustre: server umount lustre-MDT0000 complete [ 4267.986740] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4267.988156] LDISKFS-fs (dm-0): recovery complete [ 4267.990318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4269.469901] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4273.150809] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 4273.150849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 4274.286749] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4274.831115] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4277.665845] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 4278.165949] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 4280.311955] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 21:02:05 (1763085725) [ 4286.168837] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 4347.399350] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4347.401086] Lustre: Skipped 1 previous similar message [ 4347.402505] LustreError: 6516:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dcbadd0c50 x1848724970627328/t296352743435(0) o36->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:32/0 lens 66040/440 e 0 to 0 dl 1763085847 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4347.408463] LustreError: 6516:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 4407.059888] Lustre: 6518:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94dcb79e2a00 x1848724970627328/t296352743435(0) o36->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:92/0 lens 66040/440 e 0 to 0 dl 1763085907 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4410.386740] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 4410.972283] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 21:04:15 (1763085855) [ 4415.856806] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4416.721138] Lustre: Failing over lustre-MDT0000 [ 4416.929401] Lustre: server umount lustre-MDT0000 complete [ 4421.599857] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4421.604801] Lustre: Skipped 56 previous similar messages [ 4431.243483] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4431.245057] LDISKFS-fs (dm-0): recovery complete [ 4431.248163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.715650] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4436.449812] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4436.452021] Lustre: Skipped 48 previous similar messages [ 4436.632094] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4436.635351] Lustre: Skipped 12 previous similar messages [ 4436.652328] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 4436.652342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 4437.907664] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4438.518503] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4442.814769] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 21:04:47 (1763085887) [ 4450.374662] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4454.257109] Lustre: Failing over lustre-OST0000 [ 4454.280083] LustreError: 150388:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4454.282516] LustreError: 150388:0:(obd_class.h:479:obd_check_dev()) Skipped 103 previous similar messages [ 4454.302409] Lustre: server umount lustre-OST0000 complete [ 4468.493316] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 4468.495424] LDISKFS-fs (dm-2): recovery complete [ 4468.498973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4470.321376] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4470.363838] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4470.367105] Lustre: Skipped 12 previous similar messages [ 4481.932122] Lustre: Failing over lustre-OST0000 [ 4481.977717] Lustre: server umount lustre-OST0000 complete [ 4494.807678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4496.724676] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4499.865797] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4500.415797] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4534.541868] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 21:06:19 (1763085979) [ 4535.522490] Lustre: Failing over lustre-MDT0000 [ 4535.776216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4535.778751] Lustre: Skipped 2 previous similar messages [ 4535.806462] Lustre: server umount lustre-MDT0000 complete [ 4548.757760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.194200] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4554.231861] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 4554.231861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 4561.803432] Lustre: Failing over lustre-MDT0000 [ 4561.991537] Lustre: server umount lustre-MDT0000 complete [ 4574.826223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4576.294792] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4580.342958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 4580.343089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 4581.600132] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4582.181072] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4585.725585] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 21:07:10 (1763086030) [ 4597.149178] Lustre: Failing over lustre-OST0000 [ 4597.194611] Lustre: server umount lustre-OST0000 complete [ 4610.093295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4612.136044] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4615.356471] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4615.978813] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4620.053184] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 21:07:44 (1763086064) [ 4620.803546] Lustre: Failing over lustre-MDT0000 [ 4621.067479] Lustre: server umount lustre-MDT0000 complete [ 4624.503878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4624.555958] Lustre: *** cfs_fail_loc=605, val=0*** [ 4624.557467] LustreError: 157045:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0733860 failed: rc = -95 [ 4624.561469] LustreError: 157045:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 4624.564442] LustreError: 157045:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 4624.567270] LustreError: 157045:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 4624.569539] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 4624.572359] LustreError: 157045:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 4624.576685] Lustre: server umount lustre-MDT0000 complete [ 4624.578628] LustreError: 157045:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 4627.389906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4628.857026] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4631.862671] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 21:07:56 (1763086076) [ 4632.564473] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 4632.564472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 4635.178735] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4636.276145] Lustre: Failing over lustre-MDT0000 [ 4636.460465] Lustre: server umount lustre-MDT0000 complete [ 4650.489401] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4650.491701] LDISKFS-fs (dm-0): recovery complete [ 4650.495083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.821233] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4652.815769] Lustre: *** cfs_fail_loc=707, val=0*** [ 4712.216070] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 4712.226157] Lustre: 159200:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4712.229197] Lustre: 159200:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 4712.310756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 4712.310791] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 4713.664120] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4714.265137] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4717.967287] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 21:09:22 (1763086162) [ 4741.515324] LustreError: 18219:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff94dcb79f3480 x1848724971507712/t0(0) o101->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:382/0 lens 664/0 e 0 to 0 dl 1763086197 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4741.524052] LustreError: 18219:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 4752.551088] LustreError: 18219:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4752.555668] LustreError: 27101:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff94dcbe46c000 x1848724971508992/t0(0) o35->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:393/0 lens 392/0 e 0 to 0 dl 1763086208 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4757.984577] LustreError: 8402:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff94dcbffbf480 x1848724983761792/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:399/0 lens 544/0 e 0 to 0 dl 1763086214 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 4757.991332] LustreError: 8402:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 48 previous similar messages [ 4764.260049] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 21:10:08 (1763086208) [ 4787.633153] LustreError: 8409:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 4798.663112] LustreError: 8409:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 4801.847633] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 21:10:46 (1763086246) [ 4824.957452] LustreError: 18219:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff94dc884fe680 x1848724971531776/t0(0) o101->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:466/0 lens 576/0 e 0 to 0 dl 1763086281 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4824.963899] LustreError: 18219:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 4830.063151] LustreError: 18219:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4830.067948] LustreError: 18219:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff94dcb2e58000 x1848724971532544/t0(0) o101->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:471/0 lens 664/0 e 0 to 0 dl 1763086286 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4840.919130] LustreError: 112493:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4840.925554] LustreError: 112493:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff94dca2bdf480 x1848724971554048/t0(0) o101->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:521/0 lens 664/0 e 0 to 0 dl 1763086336 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4840.937525] LustreError: 112493:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 4853.099464] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 21:11:37 (1763086297) [ 4948.235801] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 21:13:12 (1763086392) [ 4971.087459] LustreError: 8813:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff94dcb7018e00 x1848724971610496/t0(0) o101->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:612/0 lens 576/0 e 0 to 0 dl 1763086427 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4971.096513] LustreError: 8813:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 4971.098763] LustreError: 8813:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 4971.101073] LustreError: 8813:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 4971.519089] LustreError: 8813:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4987.337148] LustreError: 6518:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 4987.339895] LustreError: 6518:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 4987.759077] LustreError: 6518:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4987.761435] LustreError: 6518:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 5003.177496] LustreError: 6516:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff94dcb7608a80 x1848724971629952/t0(0) o36->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:644/0 lens 504/0 e 0 to 0 dl 1763086459 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 5003.187647] LustreError: 6516:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 81 previous similar messages [ 5015.547950] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 21:14:20 (1763086460) [ 5040.185409] Lustre: DEBUG MARKER: phase 2 [ 5043.404135] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 21:14:48 (1763086488) [ 5114.081267] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 21:15:58 (1763086558) [ 5114.533936] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 5115.046490] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 21:15:59 (1763086559) [ 5116.756680] Lustre: DEBUG MARKER: Started rundbench load pid=133773 ... [ 5120.168587] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5121.670597] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 5122.291869] Lustre: Failing over lustre-MDT0000 [ 5122.491587] LustreError: 166096:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5122.493765] LustreError: 166096:0:(obd_class.h:479:obd_check_dev()) Skipped 43 previous similar messages [ 5122.522738] Lustre: server umount lustre-MDT0000 complete [ 5123.861501] LustreError: 18219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5123.867169] LustreError: 18219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 141 previous similar messages [ 5127.135495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5127.135636] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.137846] LustreError: Skipped 10 previous similar messages [ 5127.143330] Lustre: Skipped 25 previous similar messages [ 5136.570968] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5136.573126] LDISKFS-fs (dm-0): recovery complete [ 5136.578040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.615692] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5136.620252] LustreError: Skipped 8 previous similar messages [ 5136.690121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5136.692038] Lustre: Skipped 11 previous similar messages [ 5136.708125] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5136.710674] Lustre: Skipped 11 previous similar messages [ 5137.979970] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5139.213989] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5139.217636] Lustre: Skipped 6 previous similar messages [ 5141.985668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5141.989287] Lustre: Skipped 25 previous similar messages [ 5141.999359] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5142.002701] Lustre: Skipped 7 previous similar messages [ 5142.017510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 5142.017694] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 5143.273404] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5143.902227] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5148.889415] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5150.414624] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 5151.055928] Lustre: Failing over lustre-MDT0001 [ 5151.065913] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 5151.244373] Lustre: server umount lustre-MDT0001 complete [ 5165.450677] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 5165.452359] LDISKFS-fs (dm-1): recovery complete [ 5165.458525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5166.860646] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5171.041386] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:817 to 0x2c0000400:833) [ 5171.041558] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:835 to 0x280000400:865) [ 5172.272431] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5172.840655] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5177.748909] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5179.315705] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 5180.017788] Lustre: Failing over lustre-MDT0000 [ 5180.284311] Lustre: server umount lustre-MDT0000 complete [ 5194.487032] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5194.489233] LDISKFS-fs (dm-0): recovery complete [ 5194.492196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5195.901564] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5199.867546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 5199.872311] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 5201.020693] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5201.615923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5206.515479] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5208.092913] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 5208.736414] Lustre: Failing over lustre-MDT0001 [ 5208.746393] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 5208.940633] Lustre: server umount lustre-MDT0001 complete [ 5222.950848] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5222.951914] LDISKFS-fs (dm-1): recovery complete [ 5222.955426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5224.339912] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5228.889437] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1284 to 0x280000400:1313) [ 5228.889723] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1252 to 0x2c0000400:1281) [ 5230.194861] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5230.813602] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5235.751182] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5237.279383] Lustre: DEBUG MARKER: test_70b fail mds1 5 times [ 5237.904864] Lustre: Failing over lustre-MDT0000 [ 5238.149878] Lustre: server umount lustre-MDT0000 complete [ 5252.296428] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5252.298347] LDISKFS-fs (dm-0): recovery complete [ 5252.301951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5253.706225] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5257.726836] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5121) [ 5257.726862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5153) [ 5258.887259] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5259.477939] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5262.741104] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 21:18:27 (1763086707) [ 5385.836607] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5396.617839] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 5397.321982] Lustre: Failing over lustre-MDT0001 [ 5397.332192] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 5397.606882] Lustre: server umount lustre-MDT0001 complete [ 5411.921911] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5411.923488] LDISKFS-fs (dm-1): recovery complete [ 5411.928236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5413.375865] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5419.303130] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2992 to 0x280000400:3009) [ 5419.313674] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2959 to 0x2c0000400:2977) [ 5420.596697] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5421.262420] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5545.328666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5556.104820] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 5556.721916] Lustre: Failing over lustre-MDT0000 [ 5556.726442] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 5556.946665] Lustre: server umount lustre-MDT0000 complete [ 5570.790697] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5570.792070] LDISKFS-fs (dm-0): recovery complete [ 5570.795807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.175918] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5578.308436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7995 to 0x280000401:8033) [ 5578.328988] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7963 to 0x2c0000401:8001) [ 5579.511669] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5580.111175] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5600.419662] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 21:24:05 (1763087045) [ 5723.102752] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5733.884903] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 5734.520849] Lustre: Failing over lustre-MDT0001 [ 5734.545156] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 5734.661152] LustreError: 178378:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5734.664174] LustreError: 178378:0:(obd_class.h:479:obd_check_dev()) Skipped 55 previous similar messages [ 5734.720140] Lustre: server umount lustre-MDT0001 complete [ 5734.880729] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5734.880974] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5734.881166] LustreError: 18219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.881175] LustreError: 18219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 84 previous similar messages [ 5734.885535] Lustre: Skipped 22 previous similar messages [ 5734.889740] LustreError: Skipped 6 previous similar messages [ 5749.048358] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5749.050593] LDISKFS-fs (dm-1): recovery complete [ 5749.054433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5749.162268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5749.166069] Lustre: Skipped 6 previous similar messages [ 5749.176303] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5749.181819] Lustre: Skipped 6 previous similar messages [ 5750.542717] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5750.545902] Lustre: Skipped 6 previous similar messages [ 5750.624336] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5754.337856] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5754.341200] Lustre: Skipped 24 previous similar messages [ 5755.384723] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5755.387584] Lustre: Skipped 6 previous similar messages [ 5755.403060] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4911 to 0x2c0000400:4929) [ 5755.403060] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4942 to 0x280000400:4961) [ 5756.712928] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5757.379250] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5881.425872] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5892.274589] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 5892.909263] Lustre: Failing over lustre-MDT0000 [ 5892.931258] LustreError: 3653:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff94db8892ad80 x1848725011367168/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 5892.940247] LustreError: 3653:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 5893.902626] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 5898.832782] Lustre: server umount lustre-MDT0000 complete [ 5913.456055] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5913.457311] LDISKFS-fs (dm-0): recovery complete [ 5913.460382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.495376] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5913.498249] LustreError: Skipped 3 previous similar messages [ 5914.839272] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5919.826971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9033 to 0x280000401:9057) [ 5919.827101] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9002 to 0x2c0000401:9025) [ 5921.117742] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5921.772350] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5925.310565] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 21:29:29 (1763087369) [ 6048.513334] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6059.279486] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 6060.057916] Lustre: Failing over lustre-MDT0001 [ 6060.088316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6060.090600] Lustre: Skipped 4 previous similar messages [ 6060.329171] Lustre: server umount lustre-MDT0001 complete [ 6075.532853] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6075.534467] LDISKFS-fs (dm-1): recovery complete [ 6075.538721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6076.981244] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6081.015503] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5386 to 0x280000400:5409) [ 6081.015519] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5354 to 0x2c0000400:5377) [ 6082.274359] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6082.842466] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6207.112637] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6217.922174] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 6218.596796] Lustre: Failing over lustre-MDT0001 [ 6218.619133] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6218.765312] Lustre: server umount lustre-MDT0001 complete [ 6232.885821] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6232.887126] LDISKFS-fs (dm-1): recovery complete [ 6232.891607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6234.427860] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6238.199446] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5386 to 0x280000400:5441) [ 6238.199508] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5354 to 0x2c0000400:5409) [ 6239.551772] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6240.289195] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6244.170426] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 21:34:48 (1763087688) [ 6249.963437] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6251.430775] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 6252.036384] Lustre: Failing over lustre-OST0000 [ 6252.076434] Lustre: server umount lustre-OST0000 complete [ 6266.087961] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6266.090109] LDISKFS-fs (dm-2): recovery complete [ 6266.093827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6268.022913] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6271.494361] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6272.141209] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6281.011223] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6282.507790] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 6283.116323] Lustre: Failing over lustre-OST0000 [ 6283.159839] Lustre: server umount lustre-OST0000 complete [ 6297.201037] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6297.203049] LDISKFS-fs (dm-2): recovery complete [ 6297.207659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6299.415015] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6302.722479] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6303.414605] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6309.599150] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 21:35:54 (1763087754) [ 6432.644649] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6435.238439] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6446.052602] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 6446.771794] Lustre: Failing over lustre-MDT0001 [ 6446.773165] LustreError: lustre-MDT0001-osp-MDT0000: operation ldlm_enqueue to node 0@lo failed: rc = -19 [ 6446.777241] LustreError: Skipped 7 previous similar messages [ 6446.779338] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6446.785071] Lustre: Skipped 14 previous similar messages [ 6446.787351] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6446.936322] LustreError: 189746:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6446.938158] LustreError: 189746:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 6447.028329] Lustre: server umount lustre-MDT0001 complete [ 6448.270899] Lustre: Failing over lustre-MDT0000 [ 6448.501146] Lustre: server umount lustre-MDT0000 complete [ 6463.655049] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6463.657585] LDISKFS-fs (dm-1): recovery complete [ 6463.662439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6463.686804] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6463.689014] LDISKFS-fs (dm-0): recovery complete [ 6463.692200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6463.794080] LustreError: 8412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6463.800195] LustreError: 8412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 72 previous similar messages [ 6463.816059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6463.817875] Lustre: Skipped 5 previous similar messages [ 6463.830484] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6463.832464] Lustre: Skipped 5 previous similar messages [ 6464.270498] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6464.273309] Lustre: Skipped 6 previous similar messages [ 6465.340788] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6465.443611] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6466.911127] Lustre: 3651:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763087896/real 1763087896] req@ffff94dcad9f1500 x1848725026928768/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763087912 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6469.088955] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6469.090811] Lustre: Skipped 16 previous similar messages [ 6469.989204] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6469.993817] Lustre: Skipped 5 previous similar messages [ 6470.010729] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5354 to 0x2c0000400:5441) [ 6470.010779] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5386 to 0x280000400:5473) [ 6474.464948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12798 to 0x280000401:12833) [ 6474.465189] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12767 to 0x2c0000401:12801) [ 6475.918879] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6476.617293] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6477.265098] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6601.392676] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6603.950417] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6614.751793] Lustre: DEBUG MARKER: fail mds1 mds2 2 times [ 6615.430374] Lustre: Failing over lustre-MDT0000 [ 6615.637983] Lustre: server umount lustre-MDT0000 complete [ 6616.351096] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763087906/real 1763087906] req@ffff94db89797800 x1848725026930048/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763087922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6616.359485] Lustre: 3654:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6616.832580] LustreError: 145049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088062 with bad export cookie 9039276435934521563 [ 6616.834116] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6616.835679] LustreError: 145049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6616.838578] LustreError: Skipped 1 previous similar message [ 6616.838719] Lustre: Failing over lustre-MDT0001 [ 6616.987534] Lustre: server umount lustre-MDT0001 complete [ 6631.577550] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 6631.579176] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6631.579609] LDISKFS-fs (dm-1): recovery complete [ 6631.580713] LDISKFS-fs (dm-0): recovery complete [ 6631.584101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6631.585956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6631.660073] LustreError: 194283:0:(llog.c:1616:llog_backup()) MGC192.168.204.105@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6631.662480] Lustre: 194283:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.105@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6642.655746] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94db8a0c5880 x1848725037264000/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6642.661486] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 6644.135325] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6644.244856] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6648.885291] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5386 to 0x280000400:5505) [ 6648.885372] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5354 to 0x2c0000400:5473) [ 6654.453798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12798 to 0x280000401:12865) [ 6654.454381] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12767 to 0x2c0000401:12833) [ 6655.818611] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6656.517589] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6657.159287] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6660.671821] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 21:41:45 (1763088105) [ 6663.375135] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6664.491675] Lustre: Failing over lustre-MDT0000 [ 6664.693305] Lustre: server umount lustre-MDT0000 complete [ 6678.735285] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6678.736542] LDISKFS-fs (dm-0): recovery complete [ 6678.738849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.311829] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 6680.177383] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6695.701786] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 6695.712831] Lustre: 196962:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94dcbfb35880 x1848725064787328/t339302454723(339302454723) o101->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:71/0 lens 520/3488 e 0 to 0 dl 1763088151 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6695.746192] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:12865) [ 6695.746195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12798 to 0x280000401:12897) [ 6697.059286] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6697.586923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6700.975944] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 21:42:25 (1763088145) [ 6703.398320] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6704.355083] Lustre: Failing over lustre-MDT0000 [ 6704.563309] Lustre: server umount lustre-MDT0000 complete [ 6718.631094] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6718.633534] LDISKFS-fs (dm-0): recovery complete [ 6718.637445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6719.990666] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6720.279029] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 6720.281953] LustreError: 194291:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dcad9f3b80 x1848725064787328/t339302454723(339302454723) o101->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:96/0 lens 520/664 e 0 to 0 dl 1763088176 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6736.667819] Lustre: lustre-MDT0000: Client b3c485f8-b8b5-445b-b430-6dd849e35313 (at 192.168.204.5@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 6736.673984] Lustre: 194293:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94db921d7800 x1848725064787328/t339302454723(339302454723) o101->b3c485f8-b8b5-445b-b430-6dd849e35313@192.168.204.5@tcp:112/0 lens 520/3488 e 0 to 0 dl 1763088192 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6736.705027] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12899 to 0x280000401:12929) [ 6736.705036] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:12897) [ 6737.991859] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6738.547901] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6741.790198] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 21:43:06 (1763088186) [ 6743.241698] Lustre: Failing over lustre-OST0000 [ 6743.284170] Lustre: server umount lustre-OST0000 complete [ 6744.470137] Lustre: Failing over lustre-MDT0000 [ 6744.558479] Lustre: server umount lustre-MDT0000 complete [ 6757.390128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6758.849017] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6759.566896] Lustre: lustre-MDT0000: Denying connection for new client c7ac99fe-97c4-449b-b5f1-b7104579da9b (at 192.168.204.5@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 6762.999148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:12929) [ 6766.666038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6767.793502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12899 to 0x280000401:12961) [ 6768.695396] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6772.714964] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 21:43:37 (1763088217) [ 6773.083593] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6773.086715] LustreError: 174110:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dc99d31c00 x1848725038741760/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:149/0 lens 1312/4320 e 0 to 0 dl 1763088229 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6775.923872] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6776.797863] Lustre: Failing over lustre-MDT0000 [ 6777.100205] Lustre: server umount lustre-MDT0000 complete [ 6789.599109] Lustre: 194320:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763088218/real 1763088218] req@ffff94dc99d30a80 x1848725038741760/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1763088234 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6789.607067] Lustre: 194320:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [ 6791.232195] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6791.233338] LDISKFS-fs (dm-0): recovery complete [ 6791.236538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6792.609396] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6796.788279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:12993) [ 6796.788280] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:12961) [ 6797.893793] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6798.421890] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6801.944859] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 21:44:06 (1763088246) [ 6802.274793] LustreError: 174111:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94db93042680 x1848725038770688/t365072220172(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:178/0 lens 2520/4320 e 0 to 0 dl 1763088258 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6804.632658] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6807.093755] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6807.874928] Lustre: Failing over lustre-MDT0001 [ 6807.881377] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 6807.883912] Lustre: Skipped 1 previous similar message [ 6814.267180] Lustre: server umount lustre-MDT0001 complete [ 6828.079291] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6828.080880] LDISKFS-fs (dm-1): recovery complete [ 6828.084847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6829.370507] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6833.644450] Lustre: 194293:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94db9504ca80 x1848725078383872/t38654708059(0) o36->c7ac99fe-97c4-449b-b5f1-b7104579da9b@192.168.204.5@tcp:233/0 lens 560/2880 e 0 to 0 dl 1763088313 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6833.650145] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5484 to 0x2c0000400:5505) [ 6833.650187] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5516 to 0x280000400:5537) [ 6834.803574] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6835.384760] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6838.882877] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 21:44:43 (1763088283) [ 6839.226601] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6839.228279] Lustre: Skipped 1 previous similar message [ 6839.229739] LustreError: 174110:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94db9504ed80 x1848725038804224/t365072220193(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:215/0 lens 2520/4320 e 0 to 0 dl 1763088295 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6841.562202] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6844.024733] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6844.636125] Lustre: Failing over lustre-MDT0000 [ 6844.926396] Lustre: server umount lustre-MDT0000 complete [ 6858.965815] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6858.967128] LDISKFS-fs (dm-0): recovery complete [ 6858.969625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.370584] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6864.397995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13025) [ 6864.398032] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:12993) [ 6865.552364] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6866.119449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6868.119452] Lustre: Failing over lustre-MDT0001 [ 6868.223321] Lustre: server umount lustre-MDT0001 complete [ 6882.397813] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6882.399154] LDISKFS-fs (dm-1): recovery complete [ 6882.402662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6883.833456] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6887.927676] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5548 to 0x280000400:5569) [ 6887.927713] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5516 to 0x2c0000400:5537) [ 6889.038962] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6889.619487] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6893.235782] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 21:45:37 (1763088337) [ 6899.072527] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6901.565270] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6902.215104] Lustre: Failing over lustre-MDT0000 [ 6902.542343] Lustre: server umount lustre-MDT0000 complete [ 6903.793891] LustreError: 145041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088349 with bad export cookie 9039276435935486716 [ 6903.794548] Lustre: Failing over lustre-MDT0001 [ 6903.796836] LustreError: 145041:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6903.799963] LustreError: 209967:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029441:0xa:0x0].0x0 (ffff94db932b6300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6909.920290] LustreError: 209967:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff94dcadb4f800 x1848725038865920/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6909.925586] LustreError: 209967:0:(client.c:1380:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [ 6909.927848] LustreError: 209967:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6910.044562] Lustre: server umount lustre-MDT0001 complete [ 6924.458974] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6924.460202] LDISKFS-fs (dm-1): recovery complete [ 6924.464294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6924.492652] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6924.494750] LDISKFS-fs (dm-0): recovery complete [ 6924.497714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6924.543533] LustreError: 211050:0:(llog.c:1616:llog_backup()) MGC192.168.204.105@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6924.546159] Lustre: 211050:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.105@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6928.799371] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94dcadb4f100 x1848725038867328/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6930.264032] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6930.363760] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6934.009236] Lustre: 211077:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94dc8d5c1880 x1848725078449920/t47244640324(0) o36->c7ac99fe-97c4-449b-b5f1-b7104579da9b@192.168.204.5@tcp:335/0 lens 560/2880 e 0 to 0 dl 1763088415 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6934.015774] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5548 to 0x2c0000400:5569) [ 6934.023910] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5580 to 0x280000400:5601) [ 6934.051592] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:13025) [ 6934.051684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13057) [ 6935.244286] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6935.798384] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6936.302107] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6939.923547] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 21:46:24 (1763088384) [ 6940.279068] LustreError: 211076:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dc99d30a80 x1848725078482688/t51539607620(0) o36->c7ac99fe-97c4-449b-b5f1-b7104579da9b@192.168.204.5@tcp:316/0 lens 560/448 e 0 to 0 dl 1763088396 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6940.285612] LustreError: 211076:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 6945.717783] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6946.357043] Lustre: Failing over lustre-MDT0000 [ 6946.624864] Lustre: server umount lustre-MDT0000 complete [ 6956.814544] Lustre: lustre-MDT0001: Client c7ac99fe-97c4-449b-b5f1-b7104579da9b (at 192.168.204.5@tcp) reconnecting [ 6956.816978] Lustre: Skipped 2 previous similar messages [ 6960.708652] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6960.710332] LDISKFS-fs (dm-0): recovery complete [ 6960.713664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.100591] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6966.271663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13089) [ 6966.271682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:13057) [ 6967.519559] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6968.049897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6971.577255] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 21:46:56 (1763088416) [ 6971.913059] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6971.914465] Lustre: Skipped 2 previous similar messages [ 6974.317286] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6974.901746] Lustre: Failing over lustre-MDT0001 [ 6981.105038] Lustre: server umount lustre-MDT0001 complete [ 6995.183555] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6995.185675] LDISKFS-fs (dm-1): recovery complete [ 6995.189523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6996.642459] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7000.556747] Lustre: 211076:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94db8e7db100 x1848725078512896/t51539607689(0) o36->c7ac99fe-97c4-449b-b5f1-b7104579da9b@192.168.204.5@tcp:376/0 lens 560/2880 e 0 to 0 dl 1763088456 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7000.564591] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5590 to 0x2c0000400:5633) [ 7000.564612] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5622 to 0x280000400:5665) [ 7000.565094] Lustre: 211076:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7001.718472] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7002.269108] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7005.783671] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 21:47:30 (1763088450) [ 7011.608531] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7014.095818] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7014.742476] Lustre: Failing over lustre-MDT0000 [ 7015.042552] Lustre: server umount lustre-MDT0000 complete [ 7022.351298] Lustre: lustre-MDT0001: Client c7ac99fe-97c4-449b-b5f1-b7104579da9b (at 192.168.204.5@tcp) reconnecting [ 7029.271532] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7029.273402] LDISKFS-fs (dm-0): recovery complete [ 7029.276382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7030.707871] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7034.881761] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:13089) [ 7034.881783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13121) [ 7036.009194] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7036.539242] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7038.477559] Lustre: Failing over lustre-MDT0001 [ 7038.566318] Lustre: server umount lustre-MDT0001 complete [ 7052.629383] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7052.630643] LDISKFS-fs (dm-1): recovery complete [ 7052.634719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7054.006276] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7057.911092] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5644 to 0x2c0000400:5665) [ 7057.911096] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5676 to 0x280000400:5697) [ 7059.106299] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7059.643315] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7063.102190] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 21:48:27 (1763088507) [ 7069.214809] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7071.846398] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7072.512940] Lustre: Failing over lustre-MDT0000 [ 7072.748355] LustreError: 220340:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7072.752074] LustreError: 220340:0:(obd_class.h:479:obd_check_dev()) Skipped 139 previous similar messages [ 7072.778287] Lustre: server umount lustre-MDT0000 complete [ 7073.248287] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7073.248557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7073.248779] LustreError: 211076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7073.248799] LustreError: 211076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 179 previous similar messages [ 7073.254754] Lustre: Skipped 57 previous similar messages [ 7073.273246] LustreError: Skipped 12 previous similar messages [ 7074.055974] LustreError: 6502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088519 with bad export cookie 9039276435935500352 [ 7074.058437] Lustre: Failing over lustre-MDT0001 [ 7074.063483] LustreError: 6502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7074.169230] Lustre: server umount lustre-MDT0001 complete [ 7088.816759] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7088.818015] LDISKFS-fs (dm-0): recovery complete [ 7088.821084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.825770] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7088.827035] LDISKFS-fs (dm-1): recovery complete [ 7088.831232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7093.727102] Lustre: 3653:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763088523/real 1763088523] req@ffff94dcb72ce680 x1848725039022208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763088539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7093.734303] Lustre: 3653:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7098.847644] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94dcbfb34a80 x1848725039024384/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7099.042259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7099.046268] Lustre: Skipped 17 previous similar messages [ 7099.061222] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7099.066425] Lustre: Skipped 17 previous similar messages [ 7100.607526] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7100.674696] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7104.206497] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7104.208541] Lustre: Skipped 16 previous similar messages [ 7104.289384] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7104.292261] Lustre: Skipped 58 previous similar messages [ 7105.320652] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7105.323174] Lustre: Skipped 17 previous similar messages [ 7105.329206] Lustre: 222394:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94db92030700 x1848725078578432/t60129542212(0) o36->c7ac99fe-97c4-449b-b5f1-b7104579da9b@192.168.204.5@tcp:481/0 lens 560/2880 e 0 to 0 dl 1763088561 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7105.334553] Lustre: 222394:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7105.338429] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5676 to 0x2c0000400:5697) [ 7105.338429] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5708 to 0x280000400:5729) [ 7113.452593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13153) [ 7113.452601] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:13121) [ 7114.789280] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7115.292509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7115.755556] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7119.227967] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 21:49:23 (1763088563) [ 7119.573441] LustreError: 174111:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94dcb8645180 x1848725039055872/t386547056653(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:495/0 lens 1488/4320 e 0 to 0 dl 1763088575 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7119.582091] LustreError: 174111:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 3 previous similar messages [ 7122.378201] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7123.016740] Lustre: Failing over lustre-MDT0001 [ 7123.026354] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 7123.030094] Lustre: Skipped 13 previous similar messages [ 7129.149764] Lustre: server umount lustre-MDT0001 complete [ 7143.169723] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7143.172108] LDISKFS-fs (dm-1): recovery complete [ 7143.176481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7144.630241] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7148.540220] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5740 to 0x280000400:5761) [ 7148.540257] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5708 to 0x2c0000400:5729) [ 7149.700523] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7150.199844] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7153.710285] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 21:49:58 (1763088598) [ 7156.446280] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7157.088310] Lustre: Failing over lustre-MDT0000 [ 7157.356802] Lustre: server umount lustre-MDT0000 complete [ 7171.276708] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7171.277929] LDISKFS-fs (dm-0): recovery complete [ 7171.280605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7172.518185] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7176.702134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13185) [ 7176.702160] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:13153) [ 7177.836969] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7178.342767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7181.566578] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 21:50:26 (1763088626) [ 7184.264102] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7186.798238] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7187.413577] Lustre: Failing over lustre-MDT0000 [ 7187.700115] Lustre: server umount lustre-MDT0000 complete [ 7201.406509] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7201.407786] LDISKFS-fs (dm-0): recovery complete [ 7201.410661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7202.663367] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7206.912865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13217) [ 7206.912903] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:13185) [ 7207.969881] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7208.450706] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7210.321843] Lustre: Failing over lustre-MDT0001 [ 7210.410968] Lustre: server umount lustre-MDT0001 complete [ 7224.137682] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7224.139248] LDISKFS-fs (dm-1): recovery complete [ 7224.142938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7225.486974] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7229.430373] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5740 to 0x280000400:5793) [ 7229.430426] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5708 to 0x2c0000400:5761) [ 7230.477422] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7230.952378] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7234.131066] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 21:51:18 (1763088678) [ 7234.463484] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7234.464809] Lustre: Skipped 5 previous similar messages [ 7236.692805] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7238.949926] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7239.534954] Lustre: Failing over lustre-MDT0000 [ 7239.805686] Lustre: server umount lustre-MDT0000 complete [ 7240.899820] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088686 with bad export cookie 9039276435935510712 [ 7240.900823] Lustre: Failing over lustre-MDT0001 [ 7240.901094] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7240.901100] LustreError: Skipped 11 previous similar messages [ 7240.902477] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7240.904483] LustreError: 231135:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029441:0x1f:0x0].0x0 (ffff94dcb73d8300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7246.816210] LustreError: 231135:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff94db94eb2a00 x1848725039167616/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7246.821173] LustreError: 231135:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7246.923590] Lustre: server umount lustre-MDT0001 complete [ 7261.703442] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7261.704582] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7261.705013] LDISKFS-fs (dm-0): recovery complete [ 7261.706051] LDISKFS-fs (dm-1): recovery complete [ 7261.708574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7261.711691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7261.793251] LustreError: 232232:0:(llog.c:1616:llog_backup()) MGC192.168.204.105@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7261.795621] Lustre: 232232:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.204.105@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7266.271503] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94db936a6a00 x1848725039168768/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7267.750793] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7267.879577] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7272.502790] Lustre: 232249:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94dcbc15b480 x1848725078654464/t73014444034(0) o36->c7ac99fe-97c4-449b-b5f1-b7104579da9b@192.168.204.5@tcp:673/0 lens 496/2888 e 0 to 0 dl 1763088753 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 7272.508178] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5740 to 0x280000400:5825) [ 7272.508188] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5708 to 0x2c0000400:5793) [ 7272.513187] Lustre: 232249:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7272.557564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13249) [ 7272.557629] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:13217) [ 7273.608234] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7274.084195] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7274.529317] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7277.644044] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 21:52:02 (1763088722) [ 7280.571989] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7281.177222] Lustre: Failing over lustre-MDT0000 [ 7281.393698] Lustre: server umount lustre-MDT0000 complete [ 7294.222405] Lustre: lustre-MDT0001: Client c7ac99fe-97c4-449b-b5f1-b7104579da9b (at 192.168.204.5@tcp) reconnecting [ 7295.013657] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7295.014827] LDISKFS-fs (dm-0): recovery complete [ 7295.017075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.276635] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7300.606900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13281) [ 7300.606925] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:13249) [ 7301.643777] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7302.123687] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7305.264940] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 21:52:30 (1763088750) [ 7307.951712] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7308.565132] Lustre: Failing over lustre-MDT0001 [ 7308.647530] Lustre: server umount lustre-MDT0001 complete [ 7322.400446] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7322.401586] LDISKFS-fs (dm-1): recovery complete [ 7322.404715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7323.640293] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7327.747354] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5740 to 0x280000400:5857) [ 7327.747383] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5708 to 0x2c0000400:5825) [ 7328.951598] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7329.663153] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7333.522899] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 21:52:58 (1763088778) [ 7337.010798] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7339.954948] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7340.684151] Lustre: Failing over lustre-MDT0000 [ 7340.910599] Lustre: server umount lustre-MDT0000 complete [ 7354.964155] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7354.965476] LDISKFS-fs (dm-0): recovery complete [ 7354.968906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7356.337455] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7360.515333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13313) [ 7360.515381] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:13281) [ 7361.585268] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7362.122337] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7363.989766] Lustre: Failing over lustre-MDT0001 [ 7364.077787] Lustre: server umount lustre-MDT0001 complete [ 7377.996368] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7377.997519] LDISKFS-fs (dm-1): recovery complete [ 7378.000482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7379.262585] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7383.539768] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5740 to 0x280000400:5889) [ 7383.539770] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5708 to 0x2c0000400:5857) [ 7384.540973] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7385.037457] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7388.186804] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 21:53:52 (1763088832) [ 7388.518775] LustreError: 232249:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff94db936a7800 x1848725078709120/t85899345922(0) o36->c7ac99fe-97c4-449b-b5f1-b7104579da9b@192.168.204.5@tcp:9/0 lens 496/456 e 0 to 0 dl 1763088844 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 7388.524062] LustreError: 232249:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 6 previous similar messages [ 7390.835705] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7393.155079] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7393.731984] Lustre: Failing over lustre-MDT0000 [ 7393.907115] Lustre: server umount lustre-MDT0000 complete [ 7395.025681] LustreError: 6502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763088840 with bad export cookie 9039276435935518209 [ 7395.026725] Lustre: Failing over lustre-MDT0001 [ 7395.028641] LustreError: 6502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7395.136451] Lustre: server umount lustre-MDT0001 complete [ 7409.909291] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7409.911367] LDISKFS-fs (dm-0): recovery complete [ 7409.915650] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7409.917881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7409.918183] LDISKFS-fs (dm-1): recovery complete [ 7409.924580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7420.384702] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7d71f606be7ee777 [ 7421.894752] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7422.039091] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7429.407744] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5708 to 0x2c0000400:5889) [ 7429.407752] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5740 to 0x280000400:5921) [ 7434.543062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12963 to 0x280000401:13345) [ 7434.543099] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12835 to 0x2c0000401:13313) [ 7435.826302] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7436.308789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7436.750747] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7439.912442] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 21:54:44 (1763088884) [ 7440.446596] Lustre: 244108:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c7ac99fe-97c4-449b-b5f1-b7104579da9b at adminstrative request [ 7444.821966] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 21:54:49 (1763088889) [ 7446.468314] Lustre: Failing over lustre-MDT0000 [ 7446.708449] Lustre: server umount lustre-MDT0000 complete [ 7459.259617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7460.472867] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7464.463944] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13364 to 0x2c0000401:13409) [ 7464.463957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13397 to 0x280000401:13441) [ 7465.497167] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7466.014818] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7469.433767] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 21:55:14 (1763088914) [ 7473.030692] Lustre: Failing over lustre-OST0000 [ 7475.155979] Lustre: server umount lustre-OST0000 complete [ 7487.750609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7489.474468] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7492.603420] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7493.156801] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7496.759423] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 21:55:41 (1763088941) [ 7497.978773] Lustre: Failing over lustre-MDT0000 [ 7498.205645] Lustre: server umount lustre-MDT0000 complete [ 7500.427496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7501.650952] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7502.295260] Lustre: lustre-MDT0000: Denying connection for new client 728398a4-0ca9-4349-b513-3667013ecc96 (at 192.168.204.5@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7502.301942] Lustre: Skipped 1 previous similar message [ 7505.909164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13397 to 0x280000401:13473) [ 7505.909169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13364 to 0x2c0000401:13441) [ 7509.890560] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 21:55:54 (1763088954) [ 7512.407499] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7513.262681] Lustre: Failing over lustre-OST0000 [ 7513.311145] Lustre: server umount lustre-OST0000 complete [ 7527.413022] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7527.414363] LDISKFS-fs (dm-2): recovery complete [ 7527.417212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7529.152642] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7532.115771] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7532.639677] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7536.106110] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 21:56:20 (1763088980) [ 7538.626947] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7540.431233] Lustre: Failing over lustre-OST0000 [ 7540.465268] Lustre: server umount lustre-OST0000 complete [ 7554.233671] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7554.234907] LDISKFS-fs (dm-2): recovery complete [ 7554.237699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7555.581288] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.5@tcp inode [0x2000301a1:0x5:0x0] object 0x280000401:13475 extent [0-4194303]: client csum f7724674, server csum af894eaa [ 7555.934645] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7558.811328] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7559.339300] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7562.660583] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 21:56:47 (1763089007) [ 7564.863982] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7567.050502] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7568.800985] Lustre: Failing over lustre-MDT0000 [ 7569.037578] Lustre: server umount lustre-MDT0000 complete [ 7570.121907] LustreError: 145049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763089015 with bad export cookie 9039276435935544970 [ 7570.123579] Lustre: Failing over lustre-OST0000 [ 7570.125089] LustreError: 145049:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7570.166783] Lustre: server umount lustre-OST0000 complete [ 7584.045190] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7584.046402] LDISKFS-fs (dm-0): recovery complete [ 7584.048732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7595.487391] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94dcb8682d80 x1848725039450368/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7595.492221] LustreError: 3650:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 7596.568340] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7600.732136] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13364 to 0x2c0000401:13473) [ 7610.359457] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7610.360630] LDISKFS-fs (dm-2): recovery complete [ 7610.363718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7612.026228] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7617.302535] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 21:57:42 (1763089062) [ 7621.798785] Lustre: Failing over lustre-OST0000 [ 7623.874567] Lustre: server umount lustre-OST0000 complete [ 7624.967846] Lustre: Failing over lustre-MDT0000 [ 7625.213444] Lustre: server umount lustre-MDT0000 complete [ 7637.607870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7638.898216] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7643.127929] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13364 to 0x2c0000401:13505) [ 7647.357096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7649.132074] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7649.787521] Lustre: lustre-OST0000: Denying connection for new client bd6525d0-cd71-44de-9251-d034a1e3ae7e (at 192.168.204.5@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 7649.791644] Lustre: Skipped 1 previous similar message [ 7655.182959] Lustre: lustre-OST0000: Denying connection for new client bd6525d0-cd71-44de-9251-d034a1e3ae7e (at 192.168.204.5@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 7665.422588] Lustre: lustre-OST0000: Denying connection for new client bd6525d0-cd71-44de-9251-d034a1e3ae7e (at 192.168.204.5@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 7665.426748] Lustre: Skipped 1 previous similar message [ 7685.902528] Lustre: lustre-OST0000: Denying connection for new client bd6525d0-cd71-44de-9251-d034a1e3ae7e (at 192.168.204.5@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:33 [ 7685.906324] Lustre: Skipped 3 previous similar messages [ 7719.000182] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 7719.001834] Lustre: 257143:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 728398a4-0ca9-4349-b513-3667013ecc96@ [ 7719.004380] Lustre: 257143:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 7719.006333] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 7719.010821] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7719.010902] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 7719.013237] Lustre: Skipped 68 previous similar messages [ 7719.013460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13524 to 0x280000401:13545) [ 7719.015632] Lustre: Skipped 21 previous similar messages [ 7723.277334] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [ 7728.082579] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 7729.957830] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 21:59:34 (1763089174) [ 7731.261751] Lustre: Failing over lustre-OST0001 [ 7731.284072] LustreError: 258476:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7731.285690] LustreError: 258476:0:(obd_class.h:479:obd_check_dev()) Skipped 163 previous similar messages [ 7731.312858] Lustre: server umount lustre-OST0001 complete [ 7731.680605] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7731.680691] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7731.682485] LustreError: Skipped 17 previous similar messages [ 7731.685171] Lustre: Skipped 67 previous similar messages [ 7731.685457] LustreError: 166213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7731.691085] LustreError: 166213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 224 previous similar messages [ 7744.120631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7744.172831] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7744.174958] Lustre: Skipped 22 previous similar messages [ 7744.177591] Lustre: lustre-OST0001: in recovery but waiting for the first client to connect [ 7744.179393] Lustre: Skipped 22 previous similar messages [ 7745.431872] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7745.433756] Lustre: Skipped 22 previous similar messages [ 7745.849418] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7749.480434] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 21:59:54 (1763089194) [ 7750.623425] Lustre: Failing over lustre-OST0000 [ 7752.464383] Lustre: lustre-OST0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 7752.466202] Lustre: Skipped 24 previous similar messages [ 7752.698571] Lustre: server umount lustre-OST0000 complete [ 7765.248300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7766.901632] LustreError: 260215:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 7766.904101] LustreError: 260215:0:(ldlm_lib.c:2883:target_recovery_thread()) Skipped 78 previous similar messages [ 7766.998505] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7773.663122] Lustre: *** cfs_fail_loc=715, val=40*** [ 7783.189965] Lustre: lustre-OST0000: Client bd6525d0-cd71-44de-9251-d034a1e3ae7e (at 192.168.204.5@tcp) reconnected, waiting for 3 clients in recovery for 1:23 [ 7783.391150] Lustre: 3650:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763089212/real 1763089212] req@ffff94dc871cb480 x1848725039562880/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763089228 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 7783.397149] Lustre: 3650:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [ 7789.535147] Lustre: *** cfs_fail_loc=715, val=40*** [ 7789.536322] Lustre: Skipped 2 previous similar messages [ 7798.751703] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 7798.754179] Lustre: Skipped 3 previous similar messages [ 7804.895120] Lustre: *** cfs_fail_loc=715, val=40*** [ 7804.896347] Lustre: Skipped 2 previous similar messages [ 7806.943087] LustreError: 260215:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7806.945171] LustreError: 260215:0:(ldlm_lib.c:2883:target_recovery_thread()) Skipped 78 previous similar messages [ 7808.143861] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7808.649995] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7811.823777] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 22:00:56 (1763089256) [ 7813.057399] Lustre: Failing over lustre-MDT0000 [ 7813.453825] Lustre: server umount lustre-MDT0000 complete [ 7825.935066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7827.164515] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7831.524853] LustreError: 261669:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 7836.639179] Lustre: *** cfs_fail_loc=715, val=80*** [ 7836.640498] Lustre: Skipped 2 previous similar messages [ 7846.670098] Lustre: lustre-MDT0000: Client bd6525d0-cd71-44de-9251-d034a1e3ae7e (at 192.168.204.5@tcp) reconnected, waiting for 2 clients in recovery for 1:25 [ 7846.672789] Lustre: Skipped 1 previous similar message [ 7846.879655] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7853.023172] Lustre: *** cfs_fail_loc=715, val=80*** [ 7853.024297] Lustre: Skipped 2 previous similar messages [ 7862.030301] Lustre: lustre-MDT0000: Client bd6525d0-cd71-44de-9251-d034a1e3ae7e (at 192.168.204.5@tcp) reconnected, waiting for 2 clients in recovery for 1:09 [ 7863.263502] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7868.383109] Lustre: *** cfs_fail_loc=715, val=80*** [ 7878.414527] Lustre: lustre-MDT0000: Client bd6525d0-cd71-44de-9251-d034a1e3ae7e (at 192.168.204.5@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 7879.647502] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7884.767273] Lustre: *** cfs_fail_loc=715, val=80*** [ 7884.770125] Lustre: Skipped 1 previous similar message [ 7894.798574] Lustre: lustre-MDT0000: Client bd6525d0-cd71-44de-9251-d034a1e3ae7e (at 192.168.204.5@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [ 7895.007756] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7911.391485] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7911.607109] LustreError: 261669:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7911.611889] Lustre: 261669:0:(ldlm_lib.c:2929:target_recovery_thread()) too long recovery - read logs [ 7911.614932] LustreError: dumping log to /tmp/lustre-log.1763089356.261669 [ 7911.672376] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13518 to 0x2c0000401:13537) [ 7911.672427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13559 to 0x280000401:13577) [ 7912.879171] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7913.385715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7916.913831] Lustre: DEBUG MARKER: == replay-single test complete, duration 7658 sec ======== 22:02:41 (1763089361) [ 7917.465230] Lustre: DEBUG MARKER: === replay-single: start cleanup 22:02:42 (1763089362) === [ 7919.954717] Lustre: DEBUG MARKER: === replay-single: finish cleanup 22:02:44 (1763089364) === [ 7956.721362] Lustre: server umount lustre-MDT0000 complete [ 7960.627658] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763089405 with bad export cookie 9039276435935557668 [ 7960.630892] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7960.634427] LustreError: 6503:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7960.644695] LustreError: Skipped 8 previous similar messages [ 7960.759278] Lustre: server umount lustre-MDT0001 complete [ 7973.843461] Lustre: server umount lustre-OST0000 complete [ 7977.605400] Lustre: server umount lustre-OST0001 complete [ 7984.418651] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 7985.934921] Key type lgssc unregistered [ 7986.098699] LNet: 264825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7986.105366] LNetError: 264825:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7986.124577] LNet: Removed LNI 192.168.204.105@tcp [ 7986.487173] Key type .llcrypt unregistered [ 7986.490081] Key type ._llcrypt unregistered