[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 427376391 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002082] x2apic enabled [ 0.002902] Switched APIC routing to physical x2apic. [ 0.003006] kvm-guest: setup PV IPIs [ 0.005270] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006012] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007006] pid_max: default: 32768 minimum: 301 [ 0.009052] LSM: Security Framework initializing [ 0.010032] Yama: becoming mindful. [ 0.011027] SELinux: Initializing. [ 0.012048] *** VALIDATE selinux *** [ 0.019474] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024102] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028059] *** VALIDATE tmpfs *** [ 0.029434] *** VALIDATE proc *** [ 0.030156] *** VALIDATE cgroup *** [ 0.031004] *** VALIDATE cgroup2 *** [ 0.032236] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035025] Spectre V2 : User space: Vulnerable [ 0.036006] Speculative Store Bypass: Vulnerable [ 0.039373] debug: unmapping init [mem 0xffffffff9b859000-0xffffffff9b860fff] [ 0.041158] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042564] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043014] ... version: 2 [ 0.043813] ... bit width: 48 [ 0.044008] ... generic registers: 4 [ 0.045007] ... value mask: 0000ffffffffffff [ 0.046007] ... max period: 00007fffffffffff [ 0.047008] ... fixed-purpose events: 3 [ 0.047952] ... event mask: 000000070000000f [ 0.049191] rcu: Hierarchical SRCU implementation. [ 0.051167] smp: Bringing up secondary CPUs ... [ 0.052427] x86: Booting SMP configuration: [ 0.053013] .... node #0, CPUs: #1 #2 #3 [ 0.056011] smp: Brought up 1 node, 4 CPUs [ 0.058011] smpboot: Max logical packages: 1 [ 0.058941] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139439] node 0 deferred pages initialised in 81ms [ 0.143169] devtmpfs: initialized [ 0.144301] x86/mm: Memory block size: 128MB [ 0.146773] gcov: version magic: 0x41383552 [ 0.150072] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151048] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152258] pinctrl core: initialized pinctrl subsystem [ 0.153115] [ 0.153439] ************************************************************* [ 0.155006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157006] ** ** [ 0.159009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161007] ** ** [ 0.162005] ** This means that this kernel is built to expose internal ** [ 0.164005] ** IOMMU data structures, which may compromise security on ** [ 0.165004] ** your system. ** [ 0.166005] ** ** [ 0.168005] ** If you see this message and you are not debugging the ** [ 0.169005] ** kernel, report this immediately to your vendor! ** [ 0.171006] ** ** [ 0.172010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174006] ************************************************************* [ 0.176491] NET: Registered protocol family 16 [ 0.177273] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182280] cpuidle: using governor menu [ 0.184459] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186366] PCI: Using configuration type 1 for base access [ 0.187096] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193134] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.196091] cryptd: max_cpu_qlen set to 1000 [ 0.198118] ACPI: Added _OSI(Module Device) [ 0.198755] ACPI: Added _OSI(Processor Device) [ 0.199005] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199799] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202787] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206314] ACPI: Interpreter enabled [ 0.207029] ACPI: PM: (supports S0 S3 S4 S5) [ 0.207769] ACPI: Using IOAPIC for interrupt routing [ 0.209037] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210186] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.216279] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.217014] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.218007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220048] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.224236] acpiphp: Slot [2] registered [ 0.225096] acpiphp: Slot [5] registered [ 0.225827] acpiphp: Slot [6] registered [ 0.227076] acpiphp: Slot [7] registered [ 0.227979] acpiphp: Slot [8] registered [ 0.228061] acpiphp: Slot [9] registered [ 0.228842] acpiphp: Slot [10] registered [ 0.230065] acpiphp: Slot [3] registered [ 0.230795] acpiphp: Slot [4] registered [ 0.231038] acpiphp: Slot [11] registered [ 0.231711] acpiphp: Slot [12] registered [ 0.233038] acpiphp: Slot [13] registered [ 0.233727] acpiphp: Slot [14] registered [ 0.234058] acpiphp: Slot [15] registered [ 0.235051] acpiphp: Slot [16] registered [ 0.235969] acpiphp: Slot [17] registered [ 0.237061] acpiphp: Slot [18] registered [ 0.238043] acpiphp: Slot [19] registered [ 0.238789] acpiphp: Slot [20] registered [ 0.240057] acpiphp: Slot [21] registered [ 0.240720] acpiphp: Slot [22] registered [ 0.241039] acpiphp: Slot [23] registered [ 0.241743] acpiphp: Slot [24] registered [ 0.243070] acpiphp: Slot [25] registered [ 0.245061] acpiphp: Slot [26] registered [ 0.246062] acpiphp: Slot [27] registered [ 0.247098] acpiphp: Slot [28] registered [ 0.249067] acpiphp: Slot [29] registered [ 0.250043] acpiphp: Slot [30] registered [ 0.251043] acpiphp: Slot [31] registered [ 0.252073] PCI host bridge to bus 0000:00 [ 0.252909] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.254012] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.255007] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.257007] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.259013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.262017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.263167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.265741] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.267712] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.275449] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.279034] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.280007] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.282009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.283007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.284471] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.286552] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.288025] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.289319] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.295007] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.303903] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.307006] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.309868] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.313007] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.317987] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.331011] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.339077] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.346013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.354014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.372015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.386048] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.392011] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.399012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.417010] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.427504] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.437014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.441007] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.452011] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.461500] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.466010] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.469984] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.482009] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.487279] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.492957] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.499012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.511011] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.520729] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.523240] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.525229] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.528280] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.530169] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.534222] iommu: Default domain type: Passthrough [ 0.536378] SCSI subsystem initialized [ 0.537076] ACPI: bus type USB registered [ 0.537828] usbcore: registered new interface driver usbfs [ 0.539037] usbcore: registered new interface driver hub [ 0.540045] usbcore: registered new device driver usb [ 0.541129] pps_core: LinuxPPS API ver. 1 registered [ 0.543008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.546037] PTP clock support registered [ 0.548146] EDAC MC: Ver: 3.0.0 [ 0.549345] PCI: Using ACPI for IRQ routing [ 0.551756] NetLabel: Initializing [ 0.552005] NetLabel: domain hash size = 128 [ 0.552894] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.554052] NetLabel: unlabeled traffic allowed by default [ 0.556106] vgaarb: loaded [ 0.557169] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.558005] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.565262] clocksource: Switched to clocksource kvm-clock [ 0.649660] VFS: Disk quotas dquot_6.6.0 [ 0.650717] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.652580] *** VALIDATE ramfs *** [ 0.653617] *** VALIDATE hugetlbfs *** [ 0.655437] pnp: PnP ACPI init [ 0.657464] pnp: PnP ACPI: found 6 devices [ 0.671739] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.674448] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.676171] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.677801] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.679622] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.681694] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.683841] NET: Registered protocol family 2 [ 0.685679] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.689442] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.692343] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.696485] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.699123] TCP: Hash tables configured (established 65536 bind 65536) [ 0.701684] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.703983] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.706282] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.709057] NET: Registered protocol family 1 [ 0.711260] RPC: Registered named UNIX socket transport module. [ 0.713167] RPC: Registered udp transport module. [ 0.714614] RPC: Registered tcp transport module. [ 0.715849] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.717534] NET: Registered protocol family 44 [ 0.718812] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.720303] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.722023] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.723609] PCI: CLS 0 bytes, default 64 [ 0.724607] Unpacking initramfs... [ 1.943766] debug: unmapping init [mem 0xffff9e09bcc54000-0xffff9e09bffbffff] [ 1.947658] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.949606] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.952069] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.372469] Initialise system trusted keyrings [ 2.373651] Key type blacklist registered [ 2.374665] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.380704] zbud: loaded [ 2.383214] *** VALIDATE nfs *** [ 2.384137] *** VALIDATE nfs4 *** [ 2.385405] pstore: using deflate compression [ 2.387179] Platform Keyring initialized [ 2.454148] NET: Registered protocol family 38 [ 2.456029] Key type asymmetric registered [ 2.457481] Asymmetric key parser 'x509' registered [ 2.459399] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.462266] io scheduler mq-deadline registered [ 2.463173] io scheduler kyber registered [ 2.464414] io scheduler bfq registered [ 2.465931] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.467748] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.469395] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.471350] ACPI: Power Button [PWRF] [ 2.475784] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.480153] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.488953] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.493263] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.505427] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.531053] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.556535] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.559757] Non-volatile memory driver v1.3 [ 2.561218] Linux agpgart interface v0.103 [ 2.583919] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.586145] vda: detected capacity change from 0 to 74706944 [ 2.598795] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.602135] vdb: detected capacity change from 0 to 1073741824 [ 2.613371] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.615360] vdc: detected capacity change from 0 to 2621440000 [ 2.625976] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.627733] vdd: detected capacity change from 0 to 2621440000 [ 2.638471] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.640479] vde: detected capacity change from 0 to 4294967296 [ 2.651074] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.653841] vdf: detected capacity change from 0 to 4294967296 [ 2.659529] libphy: Fixed MDIO Bus: probed [ 2.663416] usbcore: registered new interface driver usbserial_generic [ 2.665250] usbserial: USB Serial support registered for generic [ 2.667133] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.670450] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.671867] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.673629] mousedev: PS/2 mouse device common for all mice [ 2.675856] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.678379] rtc_cmos 00:05: RTC can wake from S4 [ 2.680803] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.682079] rtc_cmos 00:05: registered as rtc0 [ 2.684689] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.684876] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.688862] intel_pstate: CPU model not supported [ 2.690740] hid: raw HID events driver (C) Jiri Kosina [ 2.691878] usbcore: registered new interface driver usbhid [ 2.692975] usbhid: USB HID core driver [ 2.693763] drop_monitor: Initializing network drop monitor service [ 2.694917] Initializing XFRM netlink socket [ 2.695956] NET: Registered protocol family 10 [ 2.697614] Segment Routing with IPv6 [ 2.698331] NET: Registered protocol family 17 [ 2.699290] mpls_gso: MPLS GSO support [ 2.703371] RAS: Correctable Errors collector initialized. [ 2.704374] AVX version of gcm_enc/dec engaged. [ 2.705158] AES CTR mode by8 optimization enabled [ 2.755731] sched_clock: Marking stable (2755697203, 0)->(3534340460, -778643257) [ 2.757944] registered taskstats version 1 [ 2.758959] Loading compiled-in X.509 certificates [ 2.760049] zswap: loaded using pool lzo/zbud [ 2.775583] Key type big_key registered [ 2.783218] Key type encrypted registered [ 2.784772] ima: No TPM chip found, activating TPM-bypass! [ 2.785851] ima: Allocated hash algorithm: sha1 [ 2.786822] ima: No architecture policies found [ 2.787810] evm: Initialising EVM extended attributes: [ 2.788838] evm: security.selinux [ 2.789572] evm: security.ima [ 2.790140] evm: security.capability [ 2.791189] evm: HMAC attrs: 0x1 [ 2.792596] rtc_cmos 00:05: setting system clock to 2026-08-17 02:39:02 UTC (1786934342) [ 2.796214] debug: unmapping init [mem 0xffffffff9c803000-0xffffffff9c9fffff] [ 2.797731] debug: unmapping init [mem 0xffffffff9b582000-0xffffffff9b858fff] [ 2.803104] Write protecting the kernel read-only data: 28672k [ 2.806062] debug: unmapping init [mem 0xffffffff99c03000-0xffffffff99dfffff] [ 2.808138] debug: unmapping init [mem 0xffffffff9a514000-0xffffffff9a5fffff] [ 2.828842] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.833033] systemd[1]: Detected virtualization kvm. [ 2.833833] systemd[1]: Detected architecture x86-64. [ 2.834749] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.858547] systemd[1]: No hostname configured. [ 2.859914] systemd[1]: Set hostname to . [ 2.861941] random: systemd: uninitialized urandom read (16 bytes read) [ 2.863420] systemd[1]: Initializing machine ID from random generator. [ 2.901762] random: ln: uninitialized urandom read (6 bytes read) [ 2.976718] random: systemd: uninitialized urandom read (16 bytes read) [ 2.979093] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 2.982723] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 2.986126] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.433195] device-mapper: uevent: version 1.0.3 [ 3.435317] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [[ 3.694963] random: fast init done  OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 3.985530] virtio_net virtio0 ens2: renamed from eth0 [ 4.011260] scsi host0: ata_piix [ 4.020339] scsi host1: ata_piix [ 4.021458] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.023752] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.614321] dracut-initqueue[585]: RTNETLINK answers: File exists [ 8.294920] random: crng init done [ 8.295832] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 8.715714] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.571356] printk: systemd: 26 output lines suppressed due to ratelimiting [ 9.729118] SELinux: Disabled at runtime. [ 9.777388] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.781713] systemd[1]: Detected virtualization kvm. [ 9.782667] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 10.110781] systemd[1]: initrd-switch-root.service: Succeeded. [ 10.113522] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 10.116509] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 10.119788] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 10.122466] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 10.129395] systemd[1]: Starting Journal Service... Starting Journal Service... [ 10.133256] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 10.203728] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 10.460540] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 10.646962] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 10.656593] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 10.746497] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 10.756055] EDAC sbridge: Ver: 1.1.2 [ 11.900763] Key type dns_resolver registered [ 12.124535] NFS: Registering the id_resolver key type [ 12.126328] Key type id_resolver registered [ 12.127066] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg240-server login: [ 33.225604] libcfs: loading out-of-tree module taints kernel. [ 33.244250] Key type ._llcrypt registered [ 33.245296] Key type .llcrypt registered [ 33.286731] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_hostid [ 40.600332] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 41.266217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 41.272121] alg: No test for adler32 (adler32-zlib) [ 42.227162] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 42.483831] LNet: Added LNI 192.168.202.140@tcp [8/256/0/180] [ 44.079168] Key type lgssc registered [ 44.620203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 50.358319] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 64.281186] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing load_modules_local [ 69.017824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 69.025800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 70.106803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 70.116920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 70.153551] Lustre: lustre-MDT0000: new disk, initializing [ 70.175879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 70.182595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 71.514925] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 76.396235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 76.434970] Lustre: 6478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 76.445913] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 76.447405] Lustre: Skipped 1 previous similar message [ 76.483029] Lustre: lustre-MDT0001: new disk, initializing [ 76.505431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 76.513285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 76.515927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 77.863596] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 80.093544] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 83.407401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 83.506804] Lustre: lustre-OST0000: new disk, initializing [ 83.508258] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 83.510987] Lustre: 8418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 83.530412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 85.578839] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 85.678225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 85.682895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 85.706846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 91.152869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 91.197477] Lustre: lustre-OST0001: new disk, initializing [ 91.199085] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 91.201199] Lustre: 9494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 91.218484] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 91.821893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 91.825106] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 91.841839] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 93.267483] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.126980] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 104.765361] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 106.750290] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing check_logdir /tmp/testlogs/ [ 108.266458] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing yml_node [ 109.741059] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 110.613481] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 111.495655] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 112.089559] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sun Aug 16 22:40:50 EDT 2026 [ 117.440680] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 118.031026] Lustre: DEBUG MARKER: === replay-single: start setup 22:40:56 (1786934456) === [ 119.759609] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing check_config_client /mnt/lustre [ 125.588367] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 126.723142] Lustre: 13295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 127.782857] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 129.329090] Lustre: DEBUG MARKER: === replay-single: finish setup 22:41:07 (1786934467) === [ 129.832584] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 22:41:08 (1786934468) [ 132.120160] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 132.769223] Lustre: Failing over lustre-MDT0000 [ 132.868534] Lustre: server umount lustre-MDT0000 complete [ 133.087421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 133.087615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 133.091496] Lustre: Skipped 3 previous similar messages [ 138.207508] LustreError: 8887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 138.212043] LustreError: 8887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 142.937711] LustreError: 6484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 147.000430] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 147.001918] LDISKFS-fs (dm-0): recovery complete [ 147.007278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 147.020884] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 147.087378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 148.057759] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 148.499926] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.545667] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 152.557100] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 154.405836] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 154.934975] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 158.353948] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 22:41:36 (1786934496) [ 159.044504] Lustre: Failing over lustre-OST0000 [ 159.084168] Lustre: server umount lustre-OST0000 complete [ 162.783600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 162.784337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 162.786182] LustreError: Skipped 1 previous similar message [ 162.786393] LustreError: 8398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 162.789864] Lustre: Skipped 1 previous similar message [ 162.795717] LustreError: 8398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 167.904156] LustreError: 8400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 167.908051] LustreError: 8400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 171.619634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 171.666089] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.704050] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 173.022733] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 173.022749] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 173.027729] Lustre: Skipped 3 previous similar messages [ 173.343727] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 176.801460] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 177.269357] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 180.711742] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 22:41:58 (1786934518) [ 183.122284] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 183.816868] Lustre: Failing over lustre-MDT0000 [ 183.903218] Lustre: server umount lustre-MDT0000 complete [ 187.359778] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 187.360610] LustreError: 9468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 187.364855] Lustre: Skipped 3 previous similar messages [ 187.371466] LustreError: 9468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 197.599747] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 197.607662] LustreError: 6489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 197.617996] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 197.619551] LDISKFS-fs (dm-0): recovery complete [ 197.622105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 197.651994] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 197.713895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.963802] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.684142] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 199.687266] Lustre: lustre-MDT0000: Denying connection for new client 0f81b684-4bd7-4fe5-9a17-932ebf489561 (at 192.168.202.40@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 202.720942] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 202.722720] Lustre: Skipped 1 previous similar message [ 204.889825] Lustre: lustre-MDT0000: Denying connection for new client 0f81b684-4bd7-4fe5-9a17-932ebf489561 (at 192.168.202.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 210.009574] Lustre: lustre-MDT0000: Denying connection for new client 0f81b684-4bd7-4fe5-9a17-932ebf489561 (at 192.168.202.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 215.129910] Lustre: lustre-MDT0000: Denying connection for new client 0f81b684-4bd7-4fe5-9a17-932ebf489561 (at 192.168.202.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 220.249718] Lustre: lustre-MDT0000: Denying connection for new client 0f81b684-4bd7-4fe5-9a17-932ebf489561 (at 192.168.202.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 230.489684] Lustre: lustre-MDT0000: Denying connection for new client 0f81b684-4bd7-4fe5-9a17-932ebf489561 (at 192.168.202.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 230.495989] Lustre: Skipped 1 previous similar message [ 250.970281] Lustre: lustre-MDT0000: Denying connection for new client 0f81b684-4bd7-4fe5-9a17-932ebf489561 (at 192.168.202.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 250.976463] Lustre: Skipped 3 previous similar messages [ 269.500202] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 269.502175] Lustre: 18180:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 4a99c7ab-89ca-4e73-aa1e-7fabfbbf0aa0@ [ 269.506329] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 269.510471] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 269.510567] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 269.515160] Lustre: Skipped 2 previous similar messages [ 269.526743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 269.526799] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 275.266135] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 22:43:33 (1786934613) [ 277.698361] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 278.342698] Lustre: Failing over lustre-MDT0000 [ 278.420897] Lustre: server umount lustre-MDT0000 complete [ 279.519795] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.520053] LustreError: 9468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 279.524718] Lustre: Skipped 2 previous similar messages [ 279.530890] LustreError: 9468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 292.165842] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 292.167096] LDISKFS-fs (dm-0): recovery complete [ 292.169608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 292.204228] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.270195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 293.512485] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 294.208510] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 294.211168] Lustre: lustre-MDT0000: Denying connection for new client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 294.215390] Lustre: Skipped 3 previous similar messages [ 297.443970] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 361.049678] Lustre: lustre-MDT0000: Denying connection for new client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 361.056205] Lustre: Skipped 12 previous similar messages [ 364.500117] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 364.503181] Lustre: 19901:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0f81b684-4bd7-4fe5-9a17-932ebf489561@ [ 364.506144] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 364.509472] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 364.509706] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 364.511200] Lustre: Skipped 2 previous similar messages [ 364.528913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 364.528943] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 369.587893] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 22:45:07 (1786934707) [ 371.999959] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 372.635586] Lustre: Failing over lustre-MDT0000 [ 372.727583] Lustre: server umount lustre-MDT0000 complete [ 374.240028] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 374.240348] LustreError: 8428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.245635] Lustre: Skipped 3 previous similar messages [ 374.251584] LustreError: 8428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 386.882775] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 386.884498] LDISKFS-fs (dm-0): recovery complete [ 386.888241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.924024] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 386.999407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.393709] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 391.769548] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 392.161262] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 392.173349] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 392.189884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 392.189904] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 393.908450] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 394.387106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 397.614659] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 22:45:35 (1786934735) [ 399.914611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 400.507846] Lustre: Failing over lustre-MDT0000 [ 400.695531] Lustre: server umount lustre-MDT0000 complete [ 402.399948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 402.399967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 402.403877] Lustre: Skipped 3 previous similar messages [ 414.614819] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 414.616480] LDISKFS-fs (dm-0): recovery complete [ 414.620893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 414.653680] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 416.034387] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 417.369622] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 419.834259] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 419.850626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 419.850630] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 421.551275] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 422.013018] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 425.170319] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 22:46:03 (1786934763) [ 427.601075] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 428.180872] Lustre: Failing over lustre-MDT0000 [ 428.362580] Lustre: server umount lustre-MDT0000 complete [ 430.047522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 430.047940] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 430.054137] Lustre: Skipped 3 previous similar messages [ 440.287834] LustreError: 9468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 440.291885] LustreError: 9468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 442.121321] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 442.122485] LDISKFS-fs (dm-0): recovery complete [ 442.124655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.154168] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 442.969702] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 443.463650] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 447.460441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 447.462565] Lustre: Skipped 7 previous similar messages [ 447.477169] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 447.492682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 447.492723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:193) [ 449.145674] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 449.603281] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 452.927494] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 22:46:31 (1786934791) [ 455.306235] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 455.917085] Lustre: Failing over lustre-MDT0000 [ 456.090249] Lustre: server umount lustre-MDT0000 complete [ 457.696090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 470.002934] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 470.004233] LDISKFS-fs (dm-0): recovery complete [ 470.006605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 470.036964] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 471.442843] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 475.658273] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 475.658273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 477.352224] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 477.817026] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 481.004163] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 22:46:59 (1786934819) [ 483.323277] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 483.946673] Lustre: Failing over lustre-MDT0000 [ 484.124360] Lustre: server umount lustre-MDT0000 complete [ 485.856066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 485.856086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.861909] Lustre: Skipped 6 previous similar messages [ 497.849523] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 497.851367] LDISKFS-fs (dm-0): recovery complete [ 497.854386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.165632] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 499.289658] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 499.292803] Lustre: Skipped 1 previous similar message [ 503.278415] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 503.281163] Lustre: Skipped 1 previous similar message [ 503.297001] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 503.297023] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 504.987138] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 505.456702] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 508.546494] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 22:47:26 (1786934846) [ 508.850041] Lustre: *** cfs_fail_loc=13b, val=315*** [ 508.851653] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 508.853419] LustreError: 6487:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a348b8e00 x1873736498267392/t38654705666(0) o35->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:104/0 lens 392/456 e 0 to 0 dl 1786934859 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 512.107663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 512.721134] Lustre: Failing over lustre-MDT0000 [ 512.913779] Lustre: server umount lustre-MDT0000 complete [ 513.504085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 526.669712] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 526.670993] LDISKFS-fs (dm-0): recovery complete [ 526.674292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.703843] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 526.706317] LustreError: Skipped 1 previous similar message [ 526.776252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.778283] Lustre: Skipped 4 previous similar messages [ 527.998985] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 531.937397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 531.940779] Lustre: Skipped 11 previous similar messages [ 531.950147] Lustre: 6488:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a2d0d7800 x1873736498267392/t38654705666(0) o35->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:127/0 lens 392/456 e 0 to 0 dl 1786934882 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 531.962743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 531.962763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 533.786134] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 534.312674] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 537.796363] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 22:47:56 (1786934876) [ 540.285877] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 540.932247] Lustre: Failing over lustre-MDT0000 [ 541.104128] Lustre: server umount lustre-MDT0000 complete [ 555.378511] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 555.380915] LDISKFS-fs (dm-0): recovery complete [ 555.385554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.986496] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 560.639328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 560.639388] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 562.381083] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 562.887273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 565.996302] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 22:48:24 (1786934904) [ 568.342259] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 568.666053] Lustre: *** cfs_fail_loc=114, val=0*** [ 569.620639] Lustre: Failing over lustre-MDT0000 [ 569.833983] Lustre: server umount lustre-MDT0000 complete [ 570.847905] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.848282] LustreError: 8428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.851487] Lustre: Skipped 12 previous similar messages [ 570.855854] LustreError: 8428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 60 previous similar messages [ 570.858214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 570.863498] LustreError: Skipped 1 previous similar message [ 583.710986] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 583.713863] LDISKFS-fs (dm-0): recovery complete [ 583.716890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.979881] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 586.329925] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 586.333319] Lustre: Skipped 2 previous similar messages [ 589.293441] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 589.295894] Lustre: Skipped 2 previous similar messages [ 589.311162] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 589.311187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 591.093489] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 591.558525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 594.750541] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 22:48:53 (1786934933) [ 597.015863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 597.311754] Lustre: *** cfs_fail_loc=128, val=0*** [ 598.157859] Lustre: Failing over lustre-MDT0000 [ 598.318512] Lustre: server umount lustre-MDT0000 complete [ 612.131396] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 612.132594] LDISKFS-fs (dm-0): recovery complete [ 612.135487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.168879] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.171977] LustreError: Skipped 2 previous similar messages [ 612.252733] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 613.613642] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 617.467875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 617.467881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 619.290261] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 619.786745] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 623.197224] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 22:49:21 (1786934961) [ 625.699246] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 626.359059] Lustre: Failing over lustre-MDT0000 [ 626.517670] Lustre: server umount lustre-MDT0000 complete [ 640.143462] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 640.144589] LDISKFS-fs (dm-0): recovery complete [ 640.147171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 640.241731] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 641.352075] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 645.654075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 645.654075] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 647.279955] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 647.780785] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 651.022097] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 22:49:49 (1786934989) [ 653.352332] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 653.979921] Lustre: Failing over lustre-MDT0000 [ 654.170821] Lustre: server umount lustre-MDT0000 complete [ 655.839806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 655.842327] LustreError: Skipped 2 previous similar messages [ 667.897937] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 667.899063] LDISKFS-fs (dm-0): recovery complete [ 667.901024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.995149] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 669.172646] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 673.249031] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 673.251168] Lustre: Skipped 19 previous similar messages [ 673.283687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 673.283692] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 674.894233] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 675.349510] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 678.407493] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 22:50:16 (1786935016) [ 680.777492] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 682.617411] Lustre: Failing over lustre-MDT0000 [ 682.824366] Lustre: server umount lustre-MDT0000 complete [ 696.764366] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 696.765730] LDISKFS-fs (dm-0): recovery complete [ 696.768656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.878176] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 698.215406] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 702.472278] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 702.472281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 704.313719] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 704.845894] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 713.711238] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 22:50:52 (1786935052) [ 716.137414] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 716.775928] Lustre: Failing over lustre-MDT0000 [ 716.998867] Lustre: server umount lustre-MDT0000 complete [ 717.280086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.284366] Lustre: Skipped 19 previous similar messages [ 731.085688] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 731.088033] LDISKFS-fs (dm-0): recovery complete [ 731.091307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.228711] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 732.624213] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 734.809866] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 734.813211] Lustre: Skipped 4 previous similar messages [ 736.238302] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 736.241136] Lustre: Skipped 4 previous similar messages [ 736.256735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 736.256756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 738.045279] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 738.611552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 743.974380] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 22:51:22 (1786935082) [ 746.471261] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 747.120181] Lustre: Failing over lustre-MDT0000 [ 747.341871] Lustre: server umount lustre-MDT0000 complete [ 761.442196] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 761.443683] LDISKFS-fs (dm-0): recovery complete [ 761.446177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.479297] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.482170] LustreError: Skipped 4 previous similar messages [ 761.560237] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 762.999780] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 766.975432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 766.975471] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 768.652855] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 769.126256] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 772.343467] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 22:51:50 (1786935110) [ 774.600729] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 775.174918] Lustre: Failing over lustre-MDT0000 [ 775.380576] Lustre: server umount lustre-MDT0000 complete [ 789.395288] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 789.397068] LDISKFS-fs (dm-0): recovery complete [ 789.400074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.513299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.516337] Lustre: Skipped 8 previous similar messages [ 789.527550] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 790.865167] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 794.622502] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 794.622512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 796.452176] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 797.009112] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 800.440648] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 22:52:18 (1786935138) [ 802.970750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 803.626117] Lustre: Failing over lustre-MDT0000 [ 803.852936] Lustre: server umount lustre-MDT0000 complete [ 804.832410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 804.835846] LustreError: Skipped 3 previous similar messages [ 817.795796] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 817.798590] LDISKFS-fs (dm-0): recovery complete [ 817.802785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.245726] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 823.307166] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 823.307167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 825.046439] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 825.543205] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 828.972705] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 22:52:47 (1786935167) [ 831.383199] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 832.025214] Lustre: Failing over lustre-MDT0000 [ 832.091112] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 832.291408] Lustre: server umount lustre-MDT0000 complete [ 833.504053] LustreError: 8887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 833.510863] LustreError: 8887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 129 previous similar messages [ 846.307737] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 846.308995] LDISKFS-fs (dm-0): recovery complete [ 846.311341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 846.420068] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 846.422780] Lustre: Skipped 1 previous similar message [ 847.776808] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 851.453255] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 851.453255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 853.209792] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 853.747235] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 857.136936] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 22:53:15 (1786935195) [ 859.651102] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 860.297112] Lustre: Failing over lustre-MDT0000 [ 860.561962] Lustre: server umount lustre-MDT0000 complete [ 874.535044] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 874.536768] LDISKFS-fs (dm-0): recovery complete [ 874.539939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.920290] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 880.126146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 880.126234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 881.901666] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 882.440044] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 885.884227] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 22:53:44 (1786935224) [ 888.415717] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 889.067614] Lustre: Failing over lustre-MDT0000 [ 889.265063] Lustre: server umount lustre-MDT0000 complete [ 903.378686] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 903.380335] LDISKFS-fs (dm-0): recovery complete [ 903.383299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.815297] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 908.809728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 908.809731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 910.548625] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 911.079227] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 914.350237] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 22:54:12 (1786935252) [ 916.933471] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 917.555786] Lustre: Failing over lustre-MDT0000 [ 917.665278] Lustre: server umount lustre-MDT0000 complete [ 931.443340] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 931.444419] LDISKFS-fs (dm-0): recovery complete [ 931.446656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.553612] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 931.557102] Lustre: Skipped 2 previous similar messages [ 932.826887] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 936.932565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 936.936140] Lustre: Skipped 35 previous similar messages [ 936.958975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 936.958993] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 938.615558] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 939.098404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 942.360251] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 22:54:40 (1786935280) [ 944.788280] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 945.384959] Lustre: Failing over lustre-MDT0000 [ 945.481394] Lustre: server umount lustre-MDT0000 complete [ 959.505070] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 959.506271] LDISKFS-fs (dm-0): recovery complete [ 959.508966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.996861] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 965.111067] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 965.111072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 966.982968] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 967.534577] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 970.903651] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 22:55:09 (1786935309) [ 973.388115] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 974.036075] Lustre: Failing over lustre-MDT0000 [ 974.251083] Lustre: server umount lustre-MDT0000 complete [ 975.327676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.331304] Lustre: Skipped 35 previous similar messages [ 988.294941] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 988.296139] LDISKFS-fs (dm-0): recovery complete [ 988.298479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 989.793261] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 993.772517] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 993.775274] Lustre: Skipped 8 previous similar messages [ 993.790029] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:897) [ 993.790239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 995.536482] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 996.060952] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 999.573298] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 22:55:37 (1786935337) [ 1002.162508] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1002.788979] Lustre: Failing over lustre-MDT0000 [ 1002.948399] Lustre: server umount lustre-MDT0000 complete [ 1016.810727] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1016.811933] LDISKFS-fs (dm-0): recovery complete [ 1016.814687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.165441] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1021.529928] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1021.534532] Lustre: Skipped 9 previous similar messages [ 1021.948623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1021.948629] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1023.765920] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1024.334720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1027.777459] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 22:56:06 (1786935366) [ 1030.192819] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1030.855271] Lustre: Failing over lustre-MDT0000 [ 1031.028554] Lustre: server umount lustre-MDT0000 complete [ 1045.292329] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1045.293429] LDISKFS-fs (dm-0): recovery complete [ 1045.296972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.334935] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1045.338318] LustreError: Skipped 9 previous similar messages [ 1046.891425] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1050.619772] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 1050.619807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 1052.459508] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1053.027859] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1056.334192] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 22:56:34 (1786935394) [ 1058.916182] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1059.576856] Lustre: Failing over lustre-MDT0000 [ 1059.761175] Lustre: server umount lustre-MDT0000 complete [ 1060.832239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1060.834768] LustreError: Skipped 8 previous similar messages [ 1073.896857] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1073.898120] LDISKFS-fs (dm-0): recovery complete [ 1073.900591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.013882] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1074.016283] Lustre: Skipped 4 previous similar messages [ 1075.417420] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1079.289614] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 1079.289618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:993) [ 1081.061437] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1081.578162] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1084.994864] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 22:57:03 (1786935423) [ 1087.488323] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1088.171811] Lustre: Failing over lustre-MDT0000 [ 1088.399268] Lustre: server umount lustre-MDT0000 complete [ 1102.316895] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1102.318143] LDISKFS-fs (dm-0): recovery complete [ 1102.321603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.749807] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1107.967323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1107.967337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1109.797493] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1110.350172] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1113.869172] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 22:57:32 (1786935452) [ 1116.268122] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1116.945449] Lustre: Failing over lustre-MDT0000 [ 1117.097730] Lustre: server umount lustre-MDT0000 complete [ 1131.085670] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1131.086948] LDISKFS-fs (dm-0): recovery complete [ 1131.089815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.562020] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1136.638292] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1136.638292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1138.352682] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1138.842328] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1142.160635] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 22:58:00 (1786935480) [ 1144.579898] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1145.230233] Lustre: Failing over lustre-MDT0000 [ 1145.405629] Lustre: server umount lustre-MDT0000 complete [ 1159.360292] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1159.361453] LDISKFS-fs (dm-0): recovery complete [ 1159.363662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.895528] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1164.795931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 1164.795931] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 1166.562292] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1167.038211] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1170.202548] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 22:58:28 (1786935508) [ 1171.702410] Lustre: 75064:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b at adminstrative request [ 1175.151160] Lustre: Failing over lustre-MDT0000 [ 1175.336258] Lustre: server umount lustre-MDT0000 complete [ 1188.050805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.535954] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1193.459446] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 1193.463224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1195.194547] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1195.718875] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1198.061617] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1200.983663] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1203.047712] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 22:59:01 (1786935541) [ 1203.394492] Lustre: 77083:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b at adminstrative request [ 1207.904223] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 22:59:06 (1786935546) [ 1210.327464] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1211.008129] Lustre: Failing over lustre-MDT0000 [ 1211.163543] Lustre: server umount lustre-MDT0000 complete [ 1225.027198] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1225.028540] LDISKFS-fs (dm-0): recovery complete [ 1225.031476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.444626] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1230.348628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 1230.348635] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 1232.132292] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1232.596885] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1235.767425] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 22:59:34 (1786935574) [ 1238.127547] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1238.726954] Lustre: Failing over lustre-MDT0000 [ 1238.858643] Lustre: server umount lustre-MDT0000 complete [ 1252.796519] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1252.797772] LDISKFS-fs (dm-0): recovery complete [ 1252.800205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1254.253376] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1257.975978] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1257.975978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1259.760526] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1260.314322] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1263.715853] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 23:00:02 (1786935602) [ 1266.154545] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1266.782733] Lustre: Failing over lustre-MDT0000 [ 1266.941310] Lustre: server umount lustre-MDT0000 complete [ 1280.997242] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1280.998592] LDISKFS-fs (dm-0): recovery complete [ 1281.001506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.401717] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1286.138407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1286.138408] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1287.809624] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1288.300897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1291.646564] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 23:00:29 (1786935629) [ 1294.118178] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1294.780523] Lustre: Failing over lustre-MDT0000 [ 1294.937205] Lustre: server umount lustre-MDT0000 complete [ 1308.927137] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1308.928491] LDISKFS-fs (dm-0): recovery complete [ 1308.931196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1309.039749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1309.041654] Lustre: Skipped 17 previous similar messages [ 1310.423565] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1314.299188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1314.299192] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1316.095180] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1316.628902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1320.039256] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 23:00:58 (1786935658) [ 1322.530692] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1323.173578] Lustre: Failing over lustre-MDT0000 [ 1323.336087] Lustre: server umount lustre-MDT0000 complete [ 1337.233088] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1337.234349] LDISKFS-fs (dm-0): recovery complete [ 1337.236589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.345875] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1337.348573] Lustre: Skipped 8 previous similar messages [ 1338.730541] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1342.454555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 1342.454583] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 1344.188976] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1344.758363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1348.284875] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 23:01:26 (1786935686) [ 1350.761601] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1351.437624] Lustre: Failing over lustre-MDT0000 [ 1351.577870] Lustre: server umount lustre-MDT0000 complete [ 1352.672157] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.676510] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 258 previous similar messages [ 1365.689304] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1365.690448] LDISKFS-fs (dm-0): recovery complete [ 1365.692938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.083906] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1371.139763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1371.139850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1372.756560] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1373.245858] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1376.577113] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 23:01:54 (1786935714) [ 1379.006956] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1379.658132] Lustre: Failing over lustre-MDT0000 [ 1379.849231] Lustre: server umount lustre-MDT0000 complete [ 1393.964143] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1393.965343] LDISKFS-fs (dm-0): recovery complete [ 1393.968031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.391409] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1399.295791] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1399.296099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1401.047983] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1401.597654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1404.987623] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 23:02:23 (1786935743) [ 1407.532890] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1408.189889] Lustre: Failing over lustre-MDT0000 [ 1408.330454] Lustre: server umount lustre-MDT0000 complete [ 1422.462671] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1422.464423] LDISKFS-fs (dm-0): recovery complete [ 1422.466806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1423.921822] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1427.959386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1427.959413] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1429.825775] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1430.396936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1433.703239] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 23:02:52 (1786935772) [ 1436.293458] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1436.933389] Lustre: Failing over lustre-MDT0000 [ 1437.076384] Lustre: server umount lustre-MDT0000 complete [ 1451.381566] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1451.382798] LDISKFS-fs (dm-0): recovery complete [ 1451.385716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.892332] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1456.609987] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1456.612475] Lustre: Skipped 71 previous similar messages [ 1456.635381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1456.635444] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1458.472011] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1459.076538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1462.582546] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 23:03:20 (1786935800) [ 1465.090732] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1465.794244] Lustre: Failing over lustre-MDT0000 [ 1465.994131] Lustre: server umount lustre-MDT0000 complete [ 1480.137065] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1480.138722] LDISKFS-fs (dm-0): recovery complete [ 1480.142137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.488084] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1485.299881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1485.299887] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1487.077816] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1487.590693] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1491.121391] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 23:03:49 (1786935829) [ 1493.624605] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1494.273315] Lustre: Failing over lustre-MDT0000 [ 1494.500447] Lustre: server umount lustre-MDT0000 complete [ 1495.519875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.523045] Lustre: Skipped 70 previous similar messages [ 1508.602555] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1508.603718] LDISKFS-fs (dm-0): recovery complete [ 1508.606020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1510.090166] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1513.961191] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1513.963555] Lustre: Skipped 17 previous similar messages [ 1513.980931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 1513.980931] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 1515.705586] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1516.194317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1519.493729] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 23:04:17 (1786935857) [ 1519.816970] Lustre: 98603:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b at adminstrative request [ 1524.225420] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 23:04:22 (1786935862) [ 1526.338180] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1526.961560] Lustre: Failing over lustre-MDT0000 [ 1527.128712] Lustre: server umount lustre-MDT0000 complete [ 1531.281557] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1531.282670] LDISKFS-fs (dm-0): recovery complete [ 1531.284857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.393594] Lustre: lustre-MDT0000: Aborting client recovery [ 1531.395234] LustreError: 99883:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1531.397518] Lustre: 99915:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1531.399961] Lustre: 99915:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1531.402462] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1531.404718] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1531.408126] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1531.426649] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1505) [ 1531.426690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505) [ 1532.623238] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1536.480691] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1543.767298] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 23:04:42 (1786935882) [ 1546.206892] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1546.849327] Lustre: Failing over lustre-MDT0000 [ 1547.035283] Lustre: server umount lustre-MDT0000 complete [ 1551.150675] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1551.151876] LDISKFS-fs (dm-0): recovery complete [ 1551.154487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.276921] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1551.296661] Lustre: lustre-MDT0000: Aborting client recovery [ 1551.298040] LustreError: 101742:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1551.300133] Lustre: 101775:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1551.302684] Lustre: 101775:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 1551.305057] Lustre: 101775:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1551.308318] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1551.311067] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 1551.316626] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 1551.336753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1537) [ 1551.336754] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1537) [ 1552.629266] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1556.449605] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1561.859518] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1561.861905] Lustre: Skipped 1 previous similar message [ 1563.968107] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 23:05:02 (1786935902) [ 1566.557355] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1567.167109] Lustre: Failing over lustre-MDT0000 [ 1567.259440] Lustre: server umount lustre-MDT0000 complete [ 1571.488968] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1571.490595] LDISKFS-fs (dm-0): recovery complete [ 1571.493552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.532562] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1571.535596] LustreError: Skipped 18 previous similar messages [ 1571.674677] Lustre: lustre-MDT0000: Aborting client recovery [ 1571.676143] LustreError: 103606:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1571.678564] Lustre: 103638:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1571.681734] Lustre: 103638:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 1571.685325] Lustre: 103638:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1571.691303] Lustre: 103638:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1571.695366] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1571.698598] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 1571.704214] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 1571.726822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 1571.726852] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 1573.079252] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1576.928917] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1584.329315] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 23:05:22 (1786935922) [ 1584.648912] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1584.650226] LustreError: 19367:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a28e48000 x1873736499178496/t201863462916(0) o36->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:425/0 lens 512/456 e 0 to 0 dl 1786935935 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1587.206222] Lustre: Failing over lustre-MDT0000 [ 1587.296365] Lustre: server umount lustre-MDT0000 complete [ 1590.097165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.243911] Lustre: lustre-MDT0000: Aborting client recovery [ 1590.245153] LustreError: 104925:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1590.247331] Lustre: 104957:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1590.250013] Lustre: 104957:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 1590.252059] Lustre: 104957:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1590.254684] Lustre: 104957:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1590.256725] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1590.259227] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 1590.264777] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 1590.289133] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1601) [ 1590.289142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 1591.550579] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1595.360880] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1604.455948] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1605.015395] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 23:05:43 (1786935943) [ 1607.525099] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1608.426182] Lustre: Failing over lustre-MDT0000 [ 1608.598750] Lustre: server umount lustre-MDT0000 complete [ 1610.720113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1610.722509] LustreError: Skipped 16 previous similar messages [ 1612.594299] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1612.595585] LDISKFS-fs (dm-0): recovery complete [ 1612.598093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1612.725679] Lustre: lustre-MDT0000: Aborting client recovery [ 1612.726969] LustreError: 106884:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1612.729125] Lustre: 106917:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1612.732209] Lustre: 106917:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 1612.734789] Lustre: 106917:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1612.738028] Lustre: 106917:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1612.740166] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1612.743265] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 1612.749652] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 1612.771830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 1612.772226] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1633) [ 1614.079728] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1617.890244] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1625.629279] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 23:06:03 (1786935963) [ 1633.727931] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1634.355421] Lustre: Failing over lustre-MDT0000 [ 1634.586588] Lustre: server umount lustre-MDT0000 complete [ 1648.589810] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1648.590919] LDISKFS-fs (dm-0): recovery complete [ 1648.593607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1650.032614] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1651.289788] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1651.291975] Lustre: Skipped 17 previous similar messages [ 1653.771825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 1653.771828] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 1655.565804] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1656.082584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1663.896904] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 23:06:42 (1786936002) [ 1670.022449] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1672.707023] Lustre: Failing over lustre-MDT0000 [ 1672.982371] Lustre: server umount lustre-MDT0000 complete [ 1686.990905] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1686.992073] LDISKFS-fs (dm-0): recovery complete [ 1686.995201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1688.457038] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1692.731826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 1692.731865] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 1694.437959] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1694.935655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1702.809316] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 23:07:21 (1786936041) [ 1703.473401] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1703.804153] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1703.807227] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1705.756125] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 23:07:24 (1786936044) [ 1711.981765] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1714.986800] Lustre: Failing over lustre-OST0000 [ 1715.038866] Lustre: server umount lustre-OST0000 complete [ 1729.433572] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 1729.435172] LDISKFS-fs (dm-2): recovery complete [ 1729.438734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1731.488521] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1776.708964] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 23:08:35 (1786936115) [ 1779.440504] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1780.340171] Lustre: Failing over lustre-MDT0000 [ 1780.450569] Lustre: server umount lustre-MDT0000 complete [ 1794.341734] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1794.342924] LDISKFS-fs (dm-0): recovery complete [ 1794.346408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.786738] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1799.676412] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 1799.676506] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 1801.519657] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1802.078234] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1815.638780] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 23:09:13 (1786936153) [ 1816.031117] Lustre: 116235:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786936139/real 1786936139] req@ffff9e0a3c5f6300 x1873736508431360/t0(0) o5->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1786936155 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-0-0.0' uid:0 gid:0 projid:4294967295 [ 1816.037393] LustreError: 116235:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 1816.039214] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1817.055480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 1817.282356] LustreError: 8428:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1822.687141] LustreError: 8428:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1822.689503] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnecting [ 1823.268962] LustreError: 6485:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1828.319120] LustreError: 6485:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1828.321748] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnecting [ 1828.926439] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1833.951139] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1833.953403] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnecting [ 1834.542790] LustreError: 9468:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1839.583152] LustreError: 9468:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1840.228668] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1845.727150] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1845.729343] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnecting [ 1845.731357] Lustre: Skipped 1 previous similar message [ 1851.938306] LustreError: 9468:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1851.940362] LustreError: 9468:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1856.991267] LustreError: 9468:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1856.993497] LustreError: 9468:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 1862.623215] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnecting [ 1862.628854] Lustre: Skipped 2 previous similar messages [ 1869.405317] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 1869.407966] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1874.911139] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1874.914983] LustreError: 6486:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 1878.107172] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 23:10:16 (1786936216) [ 1878.751411] LustreError: 9468:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1888.346651] Lustre: lustre-MDT0000: Export ffff9e0905865800 already connecting from 192.168.202.40@tcp [ 1893.466123] Lustre: lustre-MDT0000: Export ffff9e0905865800 already connecting from 192.168.202.40@tcp [ 1898.586137] Lustre: lustre-MDT0000: Export ffff9e0905865800 already connecting from 192.168.202.40@tcp [ 1903.705752] Lustre: lustre-MDT0000: Export ffff9e0905865800 already connecting from 192.168.202.40@tcp [ 1903.708582] Lustre: Skipped 1 previous similar message [ 1908.826313] Lustre: lustre-MDT0000: Export ffff9e0905865800 already connecting from 192.168.202.40@tcp [ 1918.791161] LustreError: 9468:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 1918.797166] Lustre: 9468:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9e0906818000 x1873736501808512/t0(0) o38->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1786936238 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 1919.065903] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnecting [ 1919.069361] Lustre: Skipped 3 previous similar messages [ 1919.071215] LustreError: 6486:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1943.642590] Lustre: lustre-MDT0000: Export ffff9e0905865800 already connecting from 192.168.202.40@tcp [ 1943.647312] Lustre: Skipped 1 previous similar message [ 1959.111222] LustreError: 6486:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 1959.115104] Lustre: 6486:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9e0a34e97100 x1873736501812736/t0(0) o38->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1786936278 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1962.524134] LustreError: 6485:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1988.186909] Lustre: lustre-MDT0000: Export ffff9e0905865800 already connecting from 192.168.202.40@tcp [ 1988.192272] Lustre: Skipped 3 previous similar messages [ 2002.567062] LustreError: 6485:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2002.569250] Lustre: 6485:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9e0a3f9a0e00 x1873736501816320/t0(0) o38->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1786936322 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2003.546655] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnecting [ 2003.552583] Lustre: Skipped 1 previous similar message [ 2003.555418] LustreError: 9468:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2029.146797] Lustre: lustre-MDT0000: Export ffff9e0905865800 already connecting from 192.168.202.40@tcp [ 2029.151978] Lustre: Skipped 2 previous similar messages [ 2043.599136] LustreError: 9468:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2043.604234] Lustre: 9468:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9e0a385d0e00 x1873736501819264/t0(0) o38->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1786936363 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2044.506266] LustreError: 117808:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2084.551151] LustreError: 117808:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2084.554493] Lustre: 117808:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9e0a28e1b480 x1873736501822336/t0(0) o38->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:0/0 lens 520/416 e 0 to 0 dl 1786936404 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2085.466226] LustreError: 19367:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2089.631096] LustreError: 19367:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout interrupted [ 2091.021125] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 23:13:49 (1786936429) [ 2093.406361] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2094.738140] Lustre: Failing over lustre-MDT0000 [ 2094.878728] Lustre: server umount lustre-MDT0000 complete [ 2095.706763] LustreError: 19367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2095.718975] LustreError: 19367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 150 previous similar messages [ 2096.607798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.612580] Lustre: Skipped 37 previous similar messages [ 2099.964377] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2099.965735] LDISKFS-fs (dm-0): recovery complete [ 2099.969243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.068548] Lustre: *** cfs_fail_loc=712, val=0*** [ 2100.070258] LustreError: 43979:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9e0a3869ca80 x1873736508577920/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2100.084145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.086324] Lustre: Skipped 16 previous similar messages [ 2100.129813] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2100.129895] Lustre: lustre-MDT0000: Aborting client recovery [ 2100.131805] Lustre: Skipped 25 previous similar messages [ 2100.133147] LustreError: 121075:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2100.136642] Lustre: 121107:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2100.139620] Lustre: 121107:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 2100.141737] Lustre: 121107:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2100.144554] Lustre: 121107:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2100.146677] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2100.149233] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2100.154372] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2100.176068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2100.176078] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2101.534541] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2105.313095] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2105.321716] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2105.323547] Lustre: Skipped 49 previous similar messages [ 2111.690652] Lustre: Failing over lustre-MDT0000 [ 2111.883080] Lustre: server umount lustre-MDT0000 complete [ 2124.659835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.219380] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2129.905728] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2129.911234] Lustre: Skipped 4 previous similar messages [ 2129.931833] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2129.931833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2131.566218] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2132.042970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2135.250599] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 23:14:33 (1786936473) [ 2135.285442] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnecting [ 2135.289608] Lustre: Skipped 2 previous similar messages [ 2138.236948] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 23:14:36 (1786936476) [ 2138.536605] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2138.538057] LustreError: 6493:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a34ecdf80 x1873736501878400/t0(0) o700->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:224/0 lens 264/248 e 0 to 0 dl 1786936489 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2156.208268] Lustre: Failing over lustre-MDT0000 [ 2156.411692] Lustre: server umount lustre-MDT0000 complete [ 2169.294847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.688858] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2174.464700] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2174.464737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2176.158870] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2176.630369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2180.372300] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 23:15:18 (1786936518) [ 2181.144785] Lustre: Failing over lustre-OST0000 [ 2181.208836] Lustre: server umount lustre-OST0000 complete [ 2194.074541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2196.329087] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2200.203540] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2200.696556] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2265.959290] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 23:16:44 (1786936604) [ 2269.068586] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2270.004490] Lustre: Failing over lustre-MDT0000 [ 2270.139184] Lustre: server umount lustre-MDT0000 complete [ 2284.584804] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2284.586288] LDISKFS-fs (dm-0): recovery complete [ 2284.589842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.641658] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2284.645504] LustreError: Skipped 8 previous similar messages [ 2286.294993] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2289.242266] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2289.248408] Lustre: Skipped 6 previous similar messages [ 2290.310420] Lustre: *** cfs_fail_loc=216, val=0*** [ 2290.310546] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2290.314435] LustreError: 127229:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2291.359645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2352.947414] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 23:18:11 (1786936691) [ 2361.158283] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 23:18:19 (1786936699) [ 2362.164308] Lustre: Failing over lustre-MDT0000 [ 2362.335399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2362.337126] Lustre: server umount lustre-MDT0000 complete [ 2362.337584] LustreError: Skipped 8 previous similar messages [ 2375.077015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.673569] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2380.260890] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2380.262523] LustreError: 128808:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a28d6c380 x1873736501996416/t0(0) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:505/0 lens 328/344 e 0 to 0 dl 1786936770 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2437.210172] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnected, waiting for 2 clients in recovery for 1:05 [ 2437.218125] Lustre: 128808:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2437.247948] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 2437.247995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 2439.541498] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2440.098368] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2443.912700] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 23:19:42 (1786936782) [ 2445.295422] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2447.815060] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2448.382388] Lustre: Failing over lustre-MDT0000 [ 2448.538576] Lustre: server umount lustre-MDT0000 complete [ 2462.199282] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2462.200580] LDISKFS-fs (dm-0): recovery complete [ 2462.203415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.531683] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2467.827778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3137) [ 2467.827799] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3105) [ 2469.493952] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2470.012170] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2473.214428] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 23:20:11 (1786936811) [ 2473.563287] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2477.184311] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2477.788113] Lustre: Failing over lustre-MDT0000 [ 2477.876412] Lustre: server umount lustre-MDT0000 complete [ 2491.774878] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2491.776160] LDISKFS-fs (dm-0): recovery complete [ 2491.778672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.196467] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2497.012707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 2497.012727] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3137) [ 2498.770127] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2499.306520] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2502.595135] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 23:20:40 (1786936840) [ 2502.904891] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2506.457295] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2507.036309] Lustre: Failing over lustre-MDT0000 [ 2507.125843] Lustre: server umount lustre-MDT0000 complete [ 2520.983169] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2520.984381] LDISKFS-fs (dm-0): recovery complete [ 2520.986650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.378023] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2526.198294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3201) [ 2526.198346] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 2530.324100] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 23:21:08 (1786936868) [ 2531.652591] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2531.654827] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2531.657079] LustreError: 16853:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a0933b050 x1873736502046464/t257698037777(0) o35->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:617/0 lens 392/456 e 0 to 0 dl 1786936882 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2532.502045] Lustre: Failing over lustre-MDT0000 [ 2532.673636] Lustre: server umount lustre-MDT0000 complete [ 2545.372758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.713052] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2550.767809] Lustre: 6488:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a385c2680 x1873736502046464/t257698037777(0) o35->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:636/0 lens 392/456 e 0 to 0 dl 1786936901 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2550.773332] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 2550.773332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3233) [ 2552.457416] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2552.966412] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2556.333948] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 23:21:34 (1786936894) [ 2556.654715] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2556.655954] LustreError: 6485:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a2d403800 x1873736502059264/t261993005072(0) o36->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:686/0 lens 504/448 e 0 to 0 dl 1786936951 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2560.186086] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2560.767471] Lustre: Failing over lustre-MDT0000 [ 2560.854325] Lustre: server umount lustre-MDT0000 complete [ 2574.876280] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2574.877516] LDISKFS-fs (dm-0): recovery complete [ 2574.881260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.312173] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2580.478070] Lustre: 117808:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a34933b80 x1873736502059264/t261993005072(0) o36->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:710/0 lens 504/2880 e 0 to 0 dl 1786936975 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2580.489832] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233) [ 2580.489931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 2582.334589] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2582.888787] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2586.120162] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 23:22:04 (1786936924) [ 2586.459255] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2586.461466] LustreError: 6486:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a28e4aa00 x1873736502072960/t266287972368(0) o36->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:716/0 lens 504/448 e 0 to 0 dl 1786936981 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2587.750331] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2590.206592] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2590.861029] Lustre: Failing over lustre-MDT0000 [ 2590.953793] Lustre: server umount lustre-MDT0000 complete [ 2605.080525] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2605.081896] LDISKFS-fs (dm-0): recovery complete [ 2605.084723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.513762] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2610.668511] Lustre: 9468:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a3f9ab800 x1873736502072960/t266287972368(0) o36->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:740/0 lens 504/2880 e 0 to 0 dl 1786937005 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2610.674683] Lustre: 9468:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 2610.678293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297) [ 2610.678303] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 2614.985400] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 23:22:33 (1786936953) [ 2615.377160] LustreError: 6485:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a3c3d9f80 x1873736502085248/t270582939664(0) o36->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:745/0 lens 504/448 e 0 to 0 dl 1786937010 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2615.389153] LustreError: 6485:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 2616.649259] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2616.650583] Lustre: Skipped 1 previous similar message [ 2619.243250] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2619.864144] Lustre: Failing over lustre-MDT0000 [ 2619.959858] Lustre: server umount lustre-MDT0000 complete [ 2633.868747] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2633.871083] LDISKFS-fs (dm-0): recovery complete [ 2633.875678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2635.301311] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2639.341850] Lustre: 19367:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a2d2d9880 x1873736502085248/t270582939664(0) o36->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:14/0 lens 504/2880 e 0 to 0 dl 1786937034 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2639.349394] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 2639.349395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3329) [ 2643.358724] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 23:23:01 (1786936981) [ 2643.722136] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2644.997340] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2644.998706] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2644.999994] Lustre: Skipped 2 previous similar messages [ 2648.628151] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2649.284354] Lustre: Failing over lustre-MDT0000 [ 2649.378202] Lustre: server umount lustre-MDT0000 complete [ 2663.360085] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2663.361466] LDISKFS-fs (dm-0): recovery complete [ 2663.364720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.836160] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2668.527416] Lustre: 16853:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a3f9a8e00 x1873736502097664/t274877906960(0) o35->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:754/0 lens 392/456 e 0 to 0 dl 1786937019 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2668.534246] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 2668.534293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 2672.981044] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 23:23:31 (1786937011) [ 2673.305214] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 2673.307040] LustreError: 117808:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a385c1c00 x1873736502108160/t279172874255(0) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:48/0 lens 664/608 e 0 to 0 dl 1786937068 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2673.314824] LustreError: 117808:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 2734.189420] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnecting [ 2734.191839] Lustre: Skipped 4 previous similar messages [ 2734.194393] Lustre: 6485:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a2d2daa00 x1873736502108160/t279172874255(0) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:108/0 lens 664/3488 e 0 to 0 dl 1786937128 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2736.686426] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 23:24:34 (1786937074) [ 2739.328723] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2739.934183] Lustre: Failing over lustre-MDT0000 [ 2740.022976] Lustre: server umount lustre-MDT0000 complete [ 2740.191584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.192438] LustreError: 117808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.196558] Lustre: Skipped 57 previous similar messages [ 2740.201201] LustreError: 117808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 170 previous similar messages [ 2753.896962] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2753.898235] LDISKFS-fs (dm-0): recovery complete [ 2753.903191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2754.008856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.010827] Lustre: Skipped 13 previous similar messages [ 2754.047364] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2754.049135] Lustre: Skipped 15 previous similar messages [ 2755.454477] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2759.142240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2759.143923] Lustre: Skipped 52 previous similar messages [ 2759.154529] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2759.158145] Lustre: Skipped 12 previous similar messages [ 2759.180875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3393) [ 2759.180905] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 2761.486115] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2762.180579] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2776.353649] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 23:25:14 (1786937114) [ 2779.639359] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2780.257690] Lustre: Failing over lustre-MDT0000 [ 2780.421689] Lustre: server umount lustre-MDT0000 complete [ 2794.615327] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2794.617321] LDISKFS-fs (dm-0): recovery complete [ 2794.619985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.030549] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2800.122959] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3393) [ 2800.122959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 2801.917363] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2802.488304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2805.047266] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2810.752504] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 23:25:49 (1786937149) [ 2822.546138] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2823.227870] Lustre: Failing over lustre-MDT0000 [ 2823.571599] Lustre: server umount lustre-MDT0000 complete [ 2837.782620] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2837.783717] LDISKFS-fs (dm-0): recovery complete [ 2837.786028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.248639] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2843.214981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 2843.215028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 2845.099403] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2845.704600] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2862.995507] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 23:26:40 (1786937200) [ 2866.493611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2867.273893] Lustre: Failing over lustre-MDT0000 [ 2867.405599] Lustre: server umount lustre-MDT0000 complete [ 2881.562450] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2881.563628] LDISKFS-fs (dm-0): recovery complete [ 2881.566672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.344713] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2887.170580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 2887.170612] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 2888.950936] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2889.520946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2892.938152] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 2893.414962] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2895.411336] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 23:27:13 (1786937233) [ 2901.232509] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 2961.292433] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2961.293891] Lustre: Skipped 1 previous similar message [ 2961.295169] LustreError: 9468:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a0932c450 x1873736504896128/t296352743435(0) o36->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:335/0 lens 66040/440 e 0 to 0 dl 1786937355 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3021.915547] Lustre: 8428:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a3c5e9180 x1873736504896128/t296352743435(0) o36->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:396/0 lens 66040/440 e 0 to 0 dl 1786937416 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3024.761082] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3025.322085] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 23:29:23 (1786937363) [ 3029.505289] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3030.298080] Lustre: Failing over lustre-MDT0000 [ 3030.496222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3030.496305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3030.498092] Lustre: Skipped 2 previous similar messages [ 3030.504091] LustreError: Skipped 11 previous similar messages [ 3030.533801] Lustre: server umount lustre-MDT0000 complete [ 3044.365418] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3044.366677] LDISKFS-fs (dm-0): recovery complete [ 3044.368912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.404857] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3044.408377] LustreError: Skipped 13 previous similar messages [ 3045.848351] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3047.513885] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3047.517977] Lustre: Skipped 13 previous similar messages [ 3050.127611] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 3050.127613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 3051.913646] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3052.438931] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3056.014557] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 23:29:54 (1786937394) [ 3062.345299] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3065.310810] Lustre: Failing over lustre-OST0000 [ 3065.313598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3065.315517] Lustre: Skipped 2 previous similar messages [ 3065.349826] Lustre: server umount lustre-OST0000 complete [ 3079.958480] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3079.960207] LDISKFS-fs (dm-2): recovery complete [ 3079.964030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3082.467521] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3094.565324] Lustre: Failing over lustre-OST0000 [ 3094.639561] Lustre: server umount lustre-OST0000 complete [ 3108.775872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3111.327647] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3115.826977] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3116.587730] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3150.720801] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 23:31:29 (1786937489) [ 3151.900874] Lustre: Failing over lustre-MDT0000 [ 3152.165930] Lustre: server umount lustre-MDT0000 complete [ 3165.657070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.516632] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3171.316990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 3171.317070] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 3178.979475] Lustre: Failing over lustre-MDT0000 [ 3179.121842] Lustre: server umount lustre-MDT0000 complete [ 3191.774711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.084153] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3196.914045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 3196.914107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 3198.961405] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3199.638279] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3204.140291] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 23:32:22 (1786937542) [ 3216.053989] Lustre: Failing over lustre-OST0000 [ 3216.140677] Lustre: server umount lustre-OST0000 complete [ 3229.232862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3231.304190] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3235.328569] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3236.042270] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3240.560698] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 23:32:58 (1786937578) [ 3241.430926] Lustre: Failing over lustre-MDT0000 [ 3241.715406] Lustre: server umount lustre-MDT0000 complete [ 3245.502042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3245.538202] Lustre: *** cfs_fail_loc=605, val=0*** [ 3245.539334] LustreError: 164132:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc07774b0 failed: rc = -95 [ 3245.542273] LustreError: 164132:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 3245.543977] LustreError: 164132:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 3245.545682] LustreError: 164132:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 3245.547474] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3245.549082] LustreError: 164132:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3245.553398] Lustre: server umount lustre-MDT0000 complete [ 3245.554498] LustreError: 164132:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3248.468736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3250.096698] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3252.785082] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 23:33:11 (1786937591) [ 3253.751618] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3253.751624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3257.007446] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3258.092536] Lustre: Failing over lustre-MDT0000 [ 3258.326492] Lustre: server umount lustre-MDT0000 complete [ 3273.050704] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3273.053040] LDISKFS-fs (dm-0): recovery complete [ 3273.058059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.742373] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3277.916310] Lustre: *** cfs_fail_loc=707, val=0*** [ 3337.342816] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 3337.360758] Lustre: 166302:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3337.364762] Lustre: 166302:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 3337.495564] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 3337.495576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 3339.909945] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3340.403661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3344.933295] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 23:34:43 (1786937683) [ 3369.705640] LustreError: 6485:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9e0a385c2300 x1873736505776512/t0(0) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:700/0 lens 664/0 e 0 to 0 dl 1786937720 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3369.718830] LustreError: 6485:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3380.751050] LustreError: 6485:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3380.755784] LustreError: 16853:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9e0a3f9a1500 x1873736505778304/t0(0) o35->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:711/0 lens 392/0 e 0 to 0 dl 1786937731 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3386.848726] LustreError: 41968:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9e0a2d754a80 x1873736510457216/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:717/0 lens 544/0 e 0 to 0 dl 1786937737 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3386.853793] LustreError: 41968:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 3392.287877] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 23:35:30 (1786937730) [ 3415.613839] LustreError: 42595:0:(tgt_handler.c:2829:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3426.639166] LustreError: 42595:0:(tgt_handler.c:2829:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3430.473828] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 23:36:08 (1786937768) [ 3453.935311] LustreError: 6486:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9e0a28c0ea00 x1873736505800960/t0(0) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:29/0 lens 576/0 e 0 to 0 dl 1786937804 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3453.945681] LustreError: 6486:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3459.047097] LustreError: 6486:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3459.051444] LustreError: 6485:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9e0a4086c700 x1873736505801728/t0(0) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:34/0 lens 664/0 e 0 to 0 dl 1786937809 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3469.831172] LustreError: 6486:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3469.839897] LustreError: 8428:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9e0a4086ca80 x1873736505823232/t0(0) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:84/0 lens 664/0 e 0 to 0 dl 1786937859 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3469.854046] LustreError: 8428:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 3482.532885] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 23:37:00 (1786937820) [ 3567.782027] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 23:38:26 (1786937906) [ 3590.770377] LustreError: 8428:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9e0a34868000 x1873736505877376/t0(0) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:166/0 lens 576/0 e 0 to 0 dl 1786937941 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3590.783119] LustreError: 8428:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 3590.787955] LustreError: 8428:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3590.794934] LustreError: 8428:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 3591.215201] LustreError: 8428:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3607.081513] LustreError: 9468:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3607.083989] LustreError: 9468:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3607.503180] LustreError: 9468:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3607.507020] LustreError: 9468:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3622.938252] LustreError: 9468:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9e0a34e80000 x1873736505895936/t0(0) o36->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:198/0 lens 504/0 e 0 to 0 dl 1786937973 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 3622.945274] LustreError: 9468:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 80 previous similar messages [ 3635.173675] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 23:39:33 (1786937973) [ 3659.973811] Lustre: DEBUG MARKER: phase 2 [ 3663.339079] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 23:40:01 (1786938001) [ 3733.705525] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 23:41:11 (1786938071) [ 3734.208923] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 3734.903847] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 23:41:13 (1786938073) [ 3736.760073] Lustre: DEBUG MARKER: Started rundbench load pid=134903 ... [ 3740.342137] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3741.887981] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 3742.598891] Lustre: Failing over lustre-MDT0000 [ 3742.886656] Lustre: server umount lustre-MDT0000 complete [ 3743.843080] LustreError: 117808:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3743.853294] LustreError: 117808:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 143 previous similar messages [ 3744.224077] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3744.230546] Lustre: Skipped 41 previous similar messages [ 3757.370493] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3757.372240] LDISKFS-fs (dm-0): recovery complete [ 3757.376590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.414644] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3757.417451] LustreError: Skipped 4 previous similar messages [ 3757.489342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3757.491256] Lustre: Skipped 11 previous similar messages [ 3757.502913] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3757.504852] Lustre: Skipped 11 previous similar messages [ 3758.819869] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3759.194342] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3759.196642] Lustre: Skipped 7 previous similar messages [ 3762.659169] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3762.661772] Lustre: Skipped 41 previous similar messages [ 3762.673269] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3762.675607] Lustre: Skipped 11 previous similar messages [ 3762.693210] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) [ 3762.693258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) [ 3764.748323] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3765.281091] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3771.172159] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3772.919127] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 3773.722835] Lustre: Failing over lustre-MDT0001 [ 3773.728611] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.40@tcp (stopping) [ 3773.987960] Lustre: server umount lustre-MDT0001 complete [ 3778.017425] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3778.023026] LustreError: Skipped 5 previous similar messages [ 3788.493841] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 3788.495827] LDISKFS-fs (dm-1): recovery complete [ 3788.501972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3789.999151] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3794.369630] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:741 to 0x2c0000400:769) [ 3794.369670] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:758 to 0x280000400:801) [ 3796.118265] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3796.596319] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3801.514694] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3802.996985] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 3803.598796] Lustre: Failing over lustre-MDT0000 [ 3803.956875] Lustre: server umount lustre-MDT0000 complete [ 3818.115574] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3818.116903] LDISKFS-fs (dm-0): recovery complete [ 3818.121170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.921228] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3823.627120] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) [ 3823.627178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) [ 3825.864181] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3826.467515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3832.018420] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3833.706989] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 3834.568733] Lustre: Failing over lustre-MDT0001 [ 3834.575896] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.40@tcp (stopping) [ 3834.705219] LustreError: 8428:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000003103f2cb ns: mdt-lustre-MDT0001_UUID lock: ffff9e0a28d21c00/0x80b3431248432663 lrc: 3/0,0 mode: PR/PR res: [0x2400007ed:0x360:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.202.40@tcp remote: 0x32ae539a8bd715f7 expref: 3 pid: 8428 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 3835.009037] Lustre: server umount lustre-MDT0001 complete [ 3849.414706] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 3849.416483] LDISKFS-fs (dm-1): recovery complete [ 3849.420267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3850.698327] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3855.213160] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1188 to 0x280000400:1217) [ 3855.213168] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1156 to 0x2c0000400:1185) [ 3857.000538] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3857.604447] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3882.614380] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 23:43:40 (1786938220) [ 4006.021764] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4016.937356] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 4017.804347] Lustre: Failing over lustre-MDT0001 [ 4017.828295] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.40@tcp (stopping) [ 4018.187575] Lustre: server umount lustre-MDT0001 complete [ 4033.551417] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4033.553666] LDISKFS-fs (dm-1): recovery complete [ 4033.560888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.152593] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4041.867837] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2306 to 0x280000400:2337) [ 4041.868043] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2274 to 0x2c0000400:2305) [ 4043.907574] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4044.597465] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4168.772769] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4179.499283] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 4180.096886] Lustre: Failing over lustre-MDT0000 [ 4180.101561] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 4184.154877] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 4184.160029] Lustre: Skipped 4 previous similar messages [ 4186.258336] Lustre: server umount lustre-MDT0000 complete [ 4200.415247] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4200.416295] LDISKFS-fs (dm-0): recovery complete [ 4200.418789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.826338] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4207.773436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7038 to 0x280000401:7073) [ 4207.773732] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6974 to 0x2c0000401:7009) [ 4209.570403] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4210.141661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4238.807181] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 23:49:36 (1786938576) [ 4361.599752] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4372.322601] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 4372.929453] Lustre: Failing over lustre-MDT0000 [ 4372.945086] LustreError: 3617:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9e0a033e1880 x1873736526472064/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 4372.950261] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 4374.496135] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.499014] Lustre: Skipped 20 previous similar messages [ 4379.212615] Lustre: server umount lustre-MDT0000 complete [ 4379.616733] LustreError: 6484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4379.620579] LustreError: 6484:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 83 previous similar messages [ 4393.999388] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4394.001833] LDISKFS-fs (dm-0): recovery complete [ 4394.006034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4394.061944] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4394.066138] LustreError: Skipped 2 previous similar messages [ 4394.177564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4394.179534] Lustre: Skipped 5 previous similar messages [ 4394.193827] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4394.197071] Lustre: Skipped 5 previous similar messages [ 4395.292741] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4395.294915] Lustre: Skipped 5 previous similar messages [ 4395.755849] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4399.586885] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4399.589926] Lustre: Skipped 20 previous similar messages [ 4401.032828] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4401.035223] Lustre: Skipped 5 previous similar messages [ 4401.050485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7543 to 0x280000401:7585) [ 4401.050509] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7479 to 0x2c0000401:7521) [ 4403.395648] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4404.132456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4529.380509] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4541.307356] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 4543.087581] Lustre: Failing over lustre-MDT0000 [ 4543.122171] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.40@tcp (stopping) [ 4543.124965] Lustre: Skipped 6 previous similar messages [ 4544.480926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4544.489195] LustreError: Skipped 3 previous similar messages [ 4549.255241] Lustre: server umount lustre-MDT0000 complete [ 4563.467982] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4563.469798] LDISKFS-fs (dm-0): recovery complete [ 4563.472409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.856232] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4569.619842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7853 to 0x280000401:7873) [ 4569.619842] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7788 to 0x2c0000401:7809) [ 4571.500828] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4572.000829] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4575.587050] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 23:55:13 (1786938913) [ 4699.723405] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4710.509579] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 4711.137726] Lustre: Failing over lustre-MDT0000 [ 4711.468363] Lustre: server umount lustre-MDT0000 complete [ 4726.895656] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4726.897347] LDISKFS-fs (dm-0): recovery complete [ 4726.900653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4728.449292] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4736.213470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9628 to 0x280000401:9665) [ 4736.214541] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9564 to 0x2c0000401:9601) [ 4738.179688] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4738.692320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4862.795579] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4873.676469] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 4874.602131] Lustre: Failing over lustre-MDT0001 [ 4874.642342] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4874.646827] Lustre: Skipped 6 previous similar messages [ 4874.836928] Lustre: server umount lustre-MDT0001 complete [ 4889.975416] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4889.977533] LDISKFS-fs (dm-1): recovery complete [ 4889.982263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4891.640960] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4895.243489] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3970 to 0x2c0000400:4001) [ 4895.243518] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4002 to 0x280000400:4033) [ 4897.613241] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4898.238275] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4901.985540] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 00:00:40 (1786939240) [ 4908.774360] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4910.374885] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 4911.289810] Lustre: Failing over lustre-OST0000 [ 4911.356139] Lustre: server umount lustre-OST0000 complete [ 4926.197061] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 4926.198782] LDISKFS-fs (dm-2): recovery complete [ 4926.202336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4928.602372] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4933.253701] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4933.837656] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4943.548692] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4945.265135] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 4946.227668] Lustre: Failing over lustre-OST0000 [ 4946.288441] Lustre: server umount lustre-OST0000 complete [ 4961.474733] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 4961.476244] LDISKFS-fs (dm-2): recovery complete [ 4961.481222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4964.413876] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4969.143925] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4969.786535] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4976.686477] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 00:01:54 (1786939314) [ 5100.590592] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5103.829274] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5114.736616] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 5115.694847] Lustre: Failing over lustre-MDT0001 [ 5115.696753] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.704260] Lustre: Skipped 16 previous similar messages [ 5115.707119] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5115.873374] Lustre: server umount lustre-MDT0001 complete [ 5117.032174] LustreError: 6486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.042237] LustreError: 6486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 71 previous similar messages [ 5117.545486] Lustre: Failing over lustre-MDT0000 [ 5123.224252] Lustre: server umount lustre-MDT0000 complete [ 5138.214090] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5138.215621] LDISKFS-fs (dm-1): recovery complete [ 5138.219051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.292614] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5138.293657] LDISKFS-fs (dm-0): recovery complete [ 5138.296297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.337201] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5138.339781] LustreError: Skipped 2 previous similar messages [ 5138.341387] LustreError: 197998:0:(llog.c:1655:llog_backup()) MGC192.168.202.140@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5138.341521] LustreError: 197999:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.140@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9e0a19f3b500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5138.343697] Lustre: 197998:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.140@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5138.427734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5138.429479] Lustre: Skipped 5 previous similar messages [ 5138.447273] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5138.448948] Lustre: Skipped 5 previous similar messages [ 5140.122612] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5140.229417] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5140.577913] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5140.579730] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5140.580461] Lustre: Skipped 6 previous similar messages [ 5140.581814] Lustre: Skipped 18 previous similar messages [ 5147.747120] Lustre: lustre-MDT0001: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 5147.751981] Lustre: Skipped 5 previous similar messages [ 5147.781316] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3970 to 0x2c0000400:4033) [ 5147.781333] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4002 to 0x280000400:4065) [ 5154.936127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11164 to 0x2c0000401:11201) [ 5154.936171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11228 to 0x280000401:11265) [ 5157.958341] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5158.525471] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5159.051499] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5162.253512] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 00:05:00 (1786939500) [ 5165.238234] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5166.384033] Lustre: Failing over lustre-MDT0000 [ 5166.593397] Lustre: server umount lustre-MDT0000 complete [ 5182.134126] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5182.135436] LDISKFS-fs (dm-0): recovery complete [ 5182.143325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.584480] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5184.199869] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5199.969185] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5199.978392] Lustre: 198787:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a28d24e00 x1873736575138432/t343597399727(343597399727) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:265/0 lens 520/3488 e 0 to 0 dl 1786939550 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5200.007033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11297) [ 5200.007045] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11164 to 0x2c0000401:11233) [ 5201.911919] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5202.395521] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5205.769356] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 00:05:44 (1786939544) [ 5208.486268] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5209.456360] Lustre: Failing over lustre-MDT0000 [ 5209.690972] Lustre: server umount lustre-MDT0000 complete [ 5210.591536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5210.593962] LustreError: Skipped 5 previous similar messages [ 5223.700394] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5223.702334] LDISKFS-fs (dm-0): recovery complete [ 5223.705658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.546473] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5224.547612] LustreError: 198006:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0901ee3100 x1873736575138432/t343597399727(343597399727) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:290/0 lens 520/664 e 0 to 0 dl 1786939575 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5225.216946] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5240.929646] Lustre: lustre-MDT0000: Client 7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b (at 192.168.202.40@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5240.933430] Lustre: 198006:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e090d3c1880 x1873736575138432/t343597399727(343597399727) o101->7e2dd74c-f20d-4c1d-b2c9-cc997b8d9b5b@192.168.202.40@tcp:306/0 lens 520/3488 e 0 to 0 dl 1786939591 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5240.960555] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11235 to 0x2c0000401:11265) [ 5240.960555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11329) [ 5242.883887] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5243.388264] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5246.724625] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 00:06:25 (1786939585) [ 5248.082562] Lustre: Failing over lustre-OST0000 [ 5248.118686] Lustre: server umount lustre-OST0000 complete [ 5249.283231] Lustre: Failing over lustre-MDT0000 [ 5249.362291] Lustre: server umount lustre-MDT0000 complete [ 5262.197336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5263.686399] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5264.398777] Lustre: lustre-MDT0000: Denying connection for new client 10fb37b3-d9ad-4988-85c1-85fd0c12fdef (at 192.168.202.40@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5267.444032] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11235 to 0x2c0000401:11297) [ 5272.718857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5276.632266] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5278.193288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11361) [ 5283.965827] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 00:07:01 (1786939621) [ 5284.947290] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5284.949866] LustreError: 8410:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e090d3c2300 x1873736544166784/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:350/0 lens 1312/4320 e 0 to 0 dl 1786939635 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5290.136413] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5291.408053] Lustre: Failing over lustre-MDT0000 [ 5291.628497] Lustre: server umount lustre-MDT0000 complete [ 5301.215221] Lustre: 198032:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786939624/real 1786939624] req@ffff9e090ebe5500 x1873736544166784/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1786939640 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5308.313043] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5308.315056] LDISKFS-fs (dm-0): recovery complete [ 5308.320337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.624889] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5314.043538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11393) [ 5314.043655] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11329) [ 5316.401967] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5317.044741] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5321.244306] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 00:07:39 (1786939659) [ 5321.699691] LustreError: 180118:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e090e821c00 x1873736544196608/t365072220172(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:387/0 lens 2520/4320 e 0 to 0 dl 1786939672 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5324.804829] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5327.702714] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5328.476634] Lustre: Failing over lustre-MDT0001 [ 5328.481334] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.40@tcp (stopping) [ 5328.484230] Lustre: Skipped 4 previous similar messages [ 5334.085499] Lustre: server umount lustre-MDT0001 complete [ 5349.909481] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5349.911199] LDISKFS-fs (dm-1): recovery complete [ 5349.918407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5351.870435] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5355.514361] Lustre: 198787:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e090f198000 x1873736579988224/t25769806061(0) o36->10fb37b3-d9ad-4988-85c1-85fd0c12fdef@192.168.202.40@tcp:446/0 lens 560/2880 e 0 to 0 dl 1786939731 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5355.518359] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4076 to 0x280000400:4097) [ 5355.525359] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4044 to 0x2c0000400:4065) [ 5357.869156] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5358.576524] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5363.337797] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 00:08:21 (1786939701) [ 5363.871828] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5363.874036] Lustre: Skipped 1 previous similar message [ 5367.106647] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5370.716399] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5371.532915] Lustre: Failing over lustre-MDT0000 [ 5371.822238] Lustre: server umount lustre-MDT0000 complete [ 5380.063142] Lustre: 208889:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786939703/real 1786939703] req@ffff9e0904d61c00 x1873736544231424/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1786939719 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5387.052605] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5387.053914] LDISKFS-fs (dm-0): recovery complete [ 5387.057218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.834048] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5392.430772] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11361) [ 5392.431058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11425) [ 5394.591185] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5395.179698] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5397.298986] Lustre: Failing over lustre-MDT0001 [ 5397.392192] Lustre: server umount lustre-MDT0001 complete [ 5413.009977] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5413.012086] LDISKFS-fs (dm-1): recovery complete [ 5413.020443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5415.100468] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5418.499275] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4108 to 0x280000400:4129) [ 5418.499632] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4076 to 0x2c0000400:4097) [ 5420.973566] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5421.673778] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5426.317924] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 00:09:24 (1786939764) [ 5426.771359] LustreError: 180118:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0906598e00 x1873736544285696/t369367187472(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:492/0 lens 2520/4320 e 0 to 0 dl 1786939777 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5426.779852] LustreError: 180118:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 5433.120372] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5436.598955] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5437.360155] Lustre: Failing over lustre-MDT0000 [ 5437.554755] Lustre: server umount lustre-MDT0000 complete [ 5439.063608] LustreError: 8417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939778 with bad export cookie 9273829803492779703 [ 5439.064705] Lustre: Failing over lustre-MDT0001 [ 5439.068941] LustreError: 8417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5445.087715] LustreError: 214529:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9e0906598700 x1873736544297600/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5445.094829] LustreError: 214529:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5445.220205] Lustre: server umount lustre-MDT0001 complete [ 5460.697620] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5460.700283] LDISKFS-fs (dm-1): recovery complete [ 5460.702422] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5460.705437] LDISKFS-fs (dm-0): recovery complete [ 5460.708678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.709025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.808925] LustreError: 215626:0:(llog.c:1655:llog_backup()) MGC192.168.202.140@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5460.812712] Lustre: 215626:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.140@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5463.519466] LustreError: 3613:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e0903972a00 x1873736544298752/t0(0) o250->MGC192.168.202.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5465.237752] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5465.336313] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5469.823369] Lustre: 216415:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a2e48c000 x1873736580058112/t34359738438(0) o36->10fb37b3-d9ad-4988-85c1-85fd0c12fdef@192.168.202.40@tcp:560/0 lens 560/2880 e 0 to 0 dl 1786939845 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5469.829067] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4140 to 0x280000400:4161) [ 5469.829128] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4108 to 0x2c0000400:4129) [ 5469.849902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11457) [ 5469.849922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11393) [ 5471.879816] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5472.420996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5472.969663] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5476.843510] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 00:10:15 (1786939815) [ 5482.953568] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5483.625616] Lustre: Failing over lustre-MDT0000 [ 5483.954291] Lustre: server umount lustre-MDT0000 complete [ 5498.521077] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5498.523268] LDISKFS-fs (dm-0): recovery complete [ 5498.526762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5500.152973] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5503.999321] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11425) [ 5503.999364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11489) [ 5505.901455] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5506.384045] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5532.761674] Lustre: lustre-MDT0001: Client 10fb37b3-d9ad-4988-85c1-85fd0c12fdef (at 192.168.202.40@tcp) reconnecting [ 5532.765072] Lustre: Skipped 2 previous similar messages [ 5532.768877] Lustre: 215637:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a2d015880 x1873736580090752/t38654705734(0) o36->10fb37b3-d9ad-4988-85c1-85fd0c12fdef@192.168.202.40@tcp:637/0 lens 560/2880 e 0 to 0 dl 1786939922 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5535.099415] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 00:11:13 (1786939873) [ 5535.433356] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5535.435390] Lustre: Skipped 2 previous similar messages [ 5537.821561] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5538.441336] Lustre: Failing over lustre-MDT0001 [ 5538.524932] Lustre: server umount lustre-MDT0001 complete [ 5552.407043] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5552.408626] LDISKFS-fs (dm-1): recovery complete [ 5552.413897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.740026] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5557.747315] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4182 to 0x280000400:4225) [ 5557.747354] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4150 to 0x2c0000400:4193) [ 5559.464937] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5560.021061] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5563.576860] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 00:11:41 (1786939901) [ 5563.908329] LustreError: 220783:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a28d26a00 x1873736580154880/t42949673028(0) o36->10fb37b3-d9ad-4988-85c1-85fd0c12fdef@192.168.202.40@tcp:629/0 lens 560/448 e 0 to 0 dl 1786939914 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5563.916977] LustreError: 220783:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 2 previous similar messages [ 5569.268781] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5571.692689] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5572.329753] Lustre: Failing over lustre-MDT0000 [ 5572.517878] Lustre: server umount lustre-MDT0000 complete [ 5586.378771] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5586.380602] LDISKFS-fs (dm-0): recovery complete [ 5586.384498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.779670] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5591.547647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11521) [ 5591.547695] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11457) [ 5593.201089] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5593.708367] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5595.594713] Lustre: Failing over lustre-MDT0001 [ 5595.679379] Lustre: server umount lustre-MDT0001 complete [ 5609.611310] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5609.612415] LDISKFS-fs (dm-1): recovery complete [ 5609.616514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5610.983501] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5615.095066] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4236 to 0x280000400:4257) [ 5615.095067] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4204 to 0x2c0000400:4225) [ 5616.845175] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5617.336083] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5620.858585] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 00:12:39 (1786939959) [ 5626.693756] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5629.161472] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5629.811383] Lustre: Failing over lustre-MDT0000 [ 5629.960901] Lustre: server umount lustre-MDT0000 complete [ 5631.144860] LustreError: 7426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939970 with bad export cookie 9273829803492793493 [ 5631.145794] Lustre: Failing over lustre-MDT0001 [ 5631.147773] LustreError: 7426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5631.247821] Lustre: server umount lustre-MDT0001 complete [ 5645.954553] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5645.956848] LDISKFS-fs (dm-0): recovery complete [ 5645.959764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.970529] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5645.973455] LDISKFS-fs (dm-1): recovery complete [ 5645.979397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5651.935320] Lustre: 3614:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786939975/real 1786939975] req@ffff9e0a2d91ed80 x1873736544463488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786939991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5651.943699] Lustre: 3614:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5656.031536] LustreError: 3613:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e0a3f8e1500 x1873736544465664/t0(0) o250->MGC192.168.202.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5657.183077] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786939980/real 1786939980] req@ffff9e0a2d91e680 x1873736544464256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786939996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5657.192070] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5657.635907] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5657.756496] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5661.343100] Lustre: 3617:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786939985/real 1786939985] req@ffff9e0a04b87800 x1873736544464512/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786940001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5661.351642] Lustre: 3617:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5662.446709] Lustre: 226808:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a2d017800 x1873736580191744/t47244640324(0) o36->10fb37b3-d9ad-4988-85c1-85fd0c12fdef@192.168.202.40@tcp:728/0 lens 560/2880 e 0 to 0 dl 1786940013 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5662.452264] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4268 to 0x280000400:4289) [ 5662.452289] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4236 to 0x2c0000400:4257) [ 5662.454595] Lustre: 226808:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages [ 5670.459332] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11489) [ 5670.459452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11553) [ 5672.605680] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5673.173866] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5673.735847] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5677.667442] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 00:13:35 (1786940015) [ 5680.674241] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5681.344344] Lustre: Failing over lustre-MDT0001 [ 5681.349926] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.40@tcp (stopping) [ 5681.353044] Lustre: Skipped 9 previous similar messages [ 5686.837418] Lustre: server umount lustre-MDT0001 complete [ 5700.959886] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5700.961405] LDISKFS-fs (dm-1): recovery complete [ 5700.965252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5702.430853] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5706.230810] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4268 to 0x2c0000400:4289) [ 5706.234275] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4321) [ 5708.015219] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5708.528891] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5712.112360] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 00:14:10 (1786940050) [ 5715.089133] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5715.788134] Lustre: Failing over lustre-MDT0000 [ 5716.106425] Lustre: server umount lustre-MDT0000 complete [ 5716.447838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5716.452517] Lustre: Skipped 61 previous similar messages [ 5721.567769] LustreError: 226807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5721.573836] LustreError: 226807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 208 previous similar messages [ 5728.735179] Lustre: 229246:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786940052/real 1786940052] req@ffff9e0a2d014e00 x1873736544528384/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1984/4320 e 0 to 1 dl 1786940068 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5728.745621] Lustre: 229246:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5730.298488] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5730.300578] LDISKFS-fs (dm-0): recovery complete [ 5730.306557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.878748] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5735.933277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11585) [ 5735.933279] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11521) [ 5737.846800] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5738.419191] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5742.043310] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 00:14:40 (1786940080) [ 5745.124196] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5747.716073] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5748.428577] Lustre: Failing over lustre-MDT0000 [ 5748.702732] Lustre: server umount lustre-MDT0000 complete [ 5758.943182] Lustre: 229246:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786940082/real 1786940082] req@ffff9e0a04b9ed80 x1873736544555648/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1984/4320 e 0 to 1 dl 1786940098 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5763.741598] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5763.743819] LDISKFS-fs (dm-0): recovery complete [ 5763.747383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5763.809260] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.813177] LustreError: Skipped 10 previous similar messages [ 5763.918446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5763.922688] Lustre: Skipped 19 previous similar messages [ 5763.959901] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5763.962688] Lustre: Skipped 19 previous similar messages [ 5765.825612] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5767.770367] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5767.775155] Lustre: Skipped 18 previous similar messages [ 5769.185702] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5769.189225] Lustre: Skipped 65 previous similar messages [ 5769.230885] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5769.233601] Lustre: Skipped 19 previous similar messages [ 5769.267170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11553) [ 5769.267537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11617) [ 5771.686123] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5772.306902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5774.372530] Lustre: Failing over lustre-MDT0001 [ 5774.463954] Lustre: server umount lustre-MDT0001 complete [ 5789.074395] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5789.075795] LDISKFS-fs (dm-1): recovery complete [ 5789.086911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.793109] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5794.292204] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4353) [ 5794.292234] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4268 to 0x2c0000400:4321) [ 5796.525291] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5797.125762] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5800.841224] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 00:15:39 (1786940139) [ 5801.259096] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5801.261950] Lustre: Skipped 5 previous similar messages [ 5804.040817] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5807.054500] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5807.820370] Lustre: Failing over lustre-MDT0000 [ 5808.150382] Lustre: server umount lustre-MDT0000 complete [ 5809.609839] LustreError: 151430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940149 with bad export cookie 9273829803492803916 [ 5809.611306] Lustre: Failing over lustre-MDT0001 [ 5809.614221] LustreError: 151430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5809.617577] LustreError: 236867:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029441:0x1f:0x0].0x0 (ffff9e0a2127ba00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5814.754664] LustreError: 236867:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9e0a4023d500 x1873736544609024/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5814.761392] LustreError: 236867:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5814.869410] Lustre: server umount lustre-MDT0001 complete [ 5829.514557] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5829.517584] LDISKFS-fs (dm-0): recovery complete [ 5829.521235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5829.529150] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5829.531588] LDISKFS-fs (dm-1): recovery complete [ 5829.538595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5835.231562] LustreError: 3613:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e0a3c5f5180 x1873736544609792/t0(0) o250->MGC192.168.202.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5835.449401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 5835.451896] LustreError: Skipped 15 previous similar messages [ 5836.985956] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5837.059695] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5840.626874] Lustre: 237981:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e0a37f82d80 x1873736580270336/t60129542146(0) o36->10fb37b3-d9ad-4988-85c1-85fd0c12fdef@192.168.202.40@tcp:177/0 lens 496/2888 e 0 to 0 dl 1786940217 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 5840.633629] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4385) [ 5840.634202] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4268 to 0x2c0000400:4353) [ 5840.634736] Lustre: 237981:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5851.768728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11585) [ 5851.768769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11649) [ 5853.990834] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5854.536544] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5855.088209] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5858.559630] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 00:16:36 (1786940196) [ 5858.960521] LustreError: 237981:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff9e0a2e48d880 x1873736580284544/t64424509442(0) o36->10fb37b3-d9ad-4988-85c1-85fd0c12fdef@192.168.202.40@tcp:169/0 lens 496/456 e 0 to 0 dl 1786940209 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 5858.969898] LustreError: 237981:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 5 previous similar messages [ 5861.867227] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5862.526663] Lustre: Failing over lustre-MDT0000 [ 5862.706606] Lustre: server umount lustre-MDT0000 complete [ 5874.268147] Lustre: lustre-MDT0001: Client 10fb37b3-d9ad-4988-85c1-85fd0c12fdef (at 192.168.202.40@tcp) reconnecting [ 5874.271915] Lustre: Skipped 1 previous similar message [ 5876.923738] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5876.925244] LDISKFS-fs (dm-0): recovery complete [ 5876.928151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5878.395770] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5882.371197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11681) [ 5882.371431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11617) [ 5884.297695] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5884.858975] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5888.559965] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 00:17:06 (1786940226) [ 5891.783075] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5892.506776] Lustre: Failing over lustre-MDT0001 [ 5892.599916] Lustre: server umount lustre-MDT0001 complete [ 5906.836268] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5906.838034] LDISKFS-fs (dm-1): recovery complete [ 5906.841844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5908.501873] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5912.059983] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4268 to 0x2c0000400:4385) [ 5912.060285] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4417) [ 5913.980502] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5914.534540] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5918.159586] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 00:17:36 (1786940256) [ 5921.011124] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5923.560869] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5924.192136] Lustre: Failing over lustre-MDT0000 [ 5924.329358] Lustre: server umount lustre-MDT0000 complete [ 5938.487070] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5938.495685] LDISKFS-fs (dm-0): recovery complete [ 5938.500752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.102659] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5943.814891] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11649) [ 5943.815178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11713) [ 5945.708077] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5946.259532] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5948.269294] Lustre: Failing over lustre-MDT0001 [ 5948.366811] Lustre: server umount lustre-MDT0001 complete [ 5962.489595] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5962.490987] LDISKFS-fs (dm-1): recovery complete [ 5962.497432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.949085] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5967.864361] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4449) [ 5967.864740] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4268 to 0x2c0000400:4417) [ 5969.622923] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5970.116892] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5973.409058] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 00:18:31 (1786940311) [ 5976.105649] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5978.485564] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5979.083945] Lustre: Failing over lustre-MDT0000 [ 5979.264989] Lustre: server umount lustre-MDT0000 complete [ 5980.507557] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940320 with bad export cookie 9273829803492811504 [ 5980.508829] Lustre: Failing over lustre-MDT0001 [ 5980.511467] LustreError: 6472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5980.610528] Lustre: server umount lustre-MDT0001 complete [ 5995.573324] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5995.575822] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5995.577058] LDISKFS-fs (dm-0): recovery complete [ 5995.578934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.580078] LDISKFS-fs (dm-1): recovery complete [ 5995.592406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.071222] Lustre: 3614:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786940322/real 1786940322] req@ffff9e0a2d28ad80 x1873736544745728/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786940338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5999.071222] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786940322/real 1786940322] req@ffff9e0a2d289c00 x1873736544745984/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786940338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5999.071246] Lustre: 3614:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5999.078337] Lustre: 3616:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6006.239548] LustreError: 3613:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e0a4086d500 x1873736544748672/t0(0) o250->MGC192.168.202.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6007.897314] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6008.043207] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6014.442500] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4481) [ 6014.442523] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4268 to 0x2c0000400:4449) [ 6020.600616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11267 to 0x280000401:11745) [ 6020.600618] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11681) [ 6022.697921] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6023.248662] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6023.770083] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6027.157680] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 00:19:25 (1786940365) [ 6027.725413] Lustre: 250562:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 10fb37b3-d9ad-4988-85c1-85fd0c12fdef at adminstrative request [ 6032.280751] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 00:19:30 (1786940370) [ 6033.983458] Lustre: Failing over lustre-MDT0000 [ 6034.152578] Lustre: server umount lustre-MDT0000 complete [ 6046.909185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.338531] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6052.361331] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11732 to 0x2c0000401:11777) [ 6052.361334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11797 to 0x280000401:11841) [ 6054.062187] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6054.539533] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6057.732875] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 00:19:56 (1786940396) [ 6061.391076] Lustre: Failing over lustre-OST0000 [ 6061.456928] Lustre: server umount lustre-OST0000 complete [ 6074.163331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6076.120428] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6079.741741] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6080.224333] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6083.600794] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 00:20:21 (1786940421) [ 6084.808792] Lustre: Failing over lustre-MDT0000 [ 6084.987285] Lustre: server umount lustre-MDT0000 complete [ 6086.624953] LustreError: 152687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940426 with bad export cookie 9273829803492824097 [ 6087.334235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.732253] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6089.438348] Lustre: lustre-MDT0000: Denying connection for new client cdf02d22-8816-4fe1-b332-86319e8d968e (at 192.168.202.40@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 6089.443409] Lustre: Skipped 2 previous similar messages [ 6092.786028] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11732 to 0x2c0000401:11809) [ 6092.786034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11797 to 0x280000401:11873) [ 6096.925967] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 00:20:35 (1786940435) [ 6099.663593] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6100.557304] Lustre: Failing over lustre-OST0000 [ 6100.596242] Lustre: server umount lustre-OST0000 complete [ 6114.613399] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6114.614819] LDISKFS-fs (dm-2): recovery complete [ 6114.620220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6116.677700] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6120.392702] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6120.885230] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6124.237640] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 00:21:02 (1786940462) [ 6126.826972] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6128.625601] Lustre: Failing over lustre-OST0000 [ 6128.661231] Lustre: server umount lustre-OST0000 complete [ 6142.707453] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6142.708807] LDISKFS-fs (dm-2): recovery complete [ 6142.711955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6144.238423] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.202.40@tcp inode [0x2000301a1:0x5:0x0] object 0x280000401:11875 extent [0-4194303]: client csum 34240956, server csum 4fc138d4 [ 6144.760334] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6148.485511] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6149.006104] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6152.502996] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 00:21:30 (1786940490) [ 6154.993866] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6157.370469] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6159.265819] Lustre: Failing over lustre-MDT0000 [ 6159.607172] Lustre: server umount lustre-MDT0000 complete [ 6160.774505] LustreError: 151430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940500 with bad export cookie 9273829803492838321 [ 6160.776806] Lustre: Failing over lustre-OST0000 [ 6160.777773] LustreError: 151430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6160.810569] Lustre: server umount lustre-OST0000 complete [ 6175.117775] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6175.119028] LDISKFS-fs (dm-0): recovery complete [ 6175.123351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6185.951761] LustreError: 3613:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e0a28e1a680 x1873736544899584/t0(0) o250->MGC192.168.202.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6187.441679] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6191.206374] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11732 to 0x2c0000401:11841) [ 6201.634965] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6201.636296] LDISKFS-fs (dm-2): recovery complete [ 6201.639408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6203.534281] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6211.086229] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 00:22:29 (1786940549) [ 6216.077062] Lustre: Failing over lustre-OST0000 [ 6216.141581] Lustre: server umount lustre-OST0000 complete [ 6217.448105] Lustre: Failing over lustre-MDT0000 [ 6217.634683] Lustre: server umount lustre-MDT0000 complete [ 6230.386671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6231.810371] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6235.640989] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11732 to 0x2c0000401:11873) [ 6240.933138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6242.897412] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6243.628190] Lustre: lustre-OST0000: Denying connection for new client 258b62b5-10a3-406b-812f-e3062cbe7499 (at 192.168.202.40@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 6249.049754] Lustre: lustre-OST0000: Denying connection for new client 258b62b5-10a3-406b-812f-e3062cbe7499 (at 192.168.202.40@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 6259.294360] Lustre: lustre-OST0000: Denying connection for new client 258b62b5-10a3-406b-812f-e3062cbe7499 (at 192.168.202.40@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 6259.303951] Lustre: Skipped 1 previous similar message [ 6279.771073] Lustre: lustre-OST0000: Denying connection for new client 258b62b5-10a3-406b-812f-e3062cbe7499 (at 192.168.202.40@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 6279.783046] Lustre: Skipped 3 previous similar messages [ 6312.500149] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6312.507531] Lustre: 264092:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client cdf02d22-8816-4fe1-b332-86319e8d968e@ [ 6312.514816] Lustre: 264092:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6312.520090] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6312.545688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11924 to 0x280000401:11945) [ 6317.154854] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 67 sec [ 6317.449615] hrtimer: interrupt took 6608263 ns [ 6326.318466] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 6329.853697] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 00:24:27 (1786940667) [ 6332.649597] Lustre: Failing over lustre-OST0000 [ 6332.701477] Lustre: server umount lustre-OST0000 complete [ 6332.895780] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6332.901741] Lustre: Skipped 53 previous similar messages [ 6332.904132] LustreError: 151432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.916589] LustreError: 151432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 206 previous similar messages [ 6347.350570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6350.383968] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6356.680229] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 00:24:54 (1786940694) [ 6358.808235] Lustre: Failing over lustre-OST0000 [ 6358.878852] Lustre: server umount lustre-OST0000 complete [ 6373.644905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6373.788202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6373.793671] Lustre: Skipped 19 previous similar messages [ 6373.799556] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6373.806574] Lustre: Skipped 19 previous similar messages [ 6374.950182] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6374.953232] Lustre: Skipped 19 previous similar messages [ 6375.219328] LustreError: 267240:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6375.224963] LustreError: 267240:0:(ldlm_lib.c:2905:target_recovery_thread()) Skipped 76 previous similar messages [ 6377.344285] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6381.023445] Lustre: *** cfs_fail_loc=715, val=40*** [ 6390.316699] Lustre: lustre-OST0000: Client 258b62b5-10a3-406b-812f-e3062cbe7499 (at 192.168.202.40@tcp) reconnected, waiting for 3 clients in recovery for 1:25 [ 6391.264277] Lustre: 3613:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786940714/real 1786940714] req@ffff9e0a2d495c00 x1873736545011840/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786940730 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 6391.277808] Lustre: 3613:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 6391.283020] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [ 6391.288542] Lustre: Skipped 1 previous similar message [ 6396.383144] Lustre: *** cfs_fail_loc=715, val=40*** [ 6396.385280] Lustre: Skipped 2 previous similar messages [ 6406.626475] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 6406.631600] Lustre: Skipped 1 previous similar message [ 6412.769139] Lustre: *** cfs_fail_loc=715, val=40*** [ 6412.774402] Lustre: Skipped 2 previous similar messages [ 6415.231088] LustreError: 267240:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6415.235902] LustreError: 267240:0:(ldlm_lib.c:2905:target_recovery_thread()) Skipped 75 previous similar messages [ 6415.259082] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6415.259307] Lustre: lustre-OST0000: Recovery over after 0:41, of 3 clients 3 recovered and 0 were evicted. [ 6415.263077] Lustre: Skipped 58 previous similar messages [ 6415.267745] Lustre: Skipped 19 previous similar messages [ 6418.221193] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6418.946236] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6423.323340] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 00:26:01 (1786940761) [ 6425.419582] Lustre: Failing over lustre-MDT0000 [ 6425.636985] Lustre: server umount lustre-MDT0000 complete [ 6440.683106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.760881] LustreError: MGC192.168.202.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6440.775346] LustreError: Skipped 8 previous similar messages [ 6445.339878] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6446.077754] LustreError: 268795:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6448.607433] Lustre: *** cfs_fail_loc=715, val=80*** [ 6448.610241] Lustre: Skipped 2 previous similar messages [ 6458.976292] Lustre: lustre-MDT0000: Client 258b62b5-10a3-406b-812f-e3062cbe7499 (at 192.168.202.40@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 6459.002345] Lustre: Skipped 1 previous similar message [ 6461.412044] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6465.503477] Lustre: *** cfs_fail_loc=715, val=80*** [ 6465.505597] Lustre: Skipped 1 previous similar message [ 6474.331416] Lustre: lustre-MDT0000: Client 258b62b5-10a3-406b-812f-e3062cbe7499 (at 192.168.202.40@tcp) reconnected, waiting for 2 clients in recovery for 1:12 [ 6477.792120] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6480.351168] Lustre: *** cfs_fail_loc=715, val=80*** [ 6480.355328] Lustre: Skipped 1 previous similar message [ 6490.716362] Lustre: lustre-MDT0000: Client 258b62b5-10a3-406b-812f-e3062cbe7499 (at 192.168.202.40@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 6494.175659] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6496.735164] Lustre: *** cfs_fail_loc=715, val=80*** [ 6496.738858] Lustre: Skipped 1 previous similar message [ 6507.103128] Lustre: lustre-MDT0000: Client 258b62b5-10a3-406b-812f-e3062cbe7499 (at 192.168.202.40@tcp) reconnected, waiting for 2 clients in recovery for 0:39 [ 6509.537112] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6525.920020] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6526.103821] LustreError: 268795:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6526.114671] Lustre: 268795:0:(ldlm_lib.c:2951:target_recovery_thread()) too long recovery - read logs [ 6526.126116] LustreError: dumping log to /tmp/lustre-log.1786940865.268795 [ 6526.276546] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11886 to 0x2c0000401:11905) [ 6526.277398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11959 to 0x280000401:11977) [ 6530.692817] Lustre: DEBUG MARKER: oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6531.943400] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6538.866760] Lustre: DEBUG MARKER: == replay-single test complete, duration 6426 sec ======== 00:27:56 (1786940876) [ 6539.891545] Lustre: DEBUG MARKER: === replay-single: start cleanup 00:27:57 (1786940877) === [ 6547.566309] Lustre: DEBUG MARKER: === replay-single: finish cleanup 00:28:05 (1786940885) === [ 6577.632301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6577.638698] LustreError: Skipped 14 previous similar messages [ 6577.640430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6577.647602] Lustre: Skipped 15 previous similar messages [ 6583.528091] Lustre: server umount lustre-MDT0000 complete [ 6589.624628] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940929 with bad export cookie 9273829803492851033 [ 6589.633274] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6589.919253] Lustre: server umount lustre-MDT0001 complete [ 6606.193364] Lustre: server umount lustre-OST0000 complete [ 6611.880858] Lustre: server umount lustre-OST0001 complete [ 6622.573849] Lustre: DEBUG MARKER: oleg240-server.virtnet: executing unload_modules_local [ 6624.906749] Key type lgssc unregistered [ 6625.152662] LNet: 272088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6625.163993] LNetError: 272088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6625.178049] LNet: Removed LNI 192.168.202.140@tcp [ 6625.872147] Key type .llcrypt unregistered [ 6625.874864] Key type ._llcrypt unregistered