[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505544226 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003266] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005025] kvm-guest: setup PV IPIs [ 0.007836] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009019] pid_max: default: 32768 minimum: 301 [ 0.011067] LSM: Security Framework initializing [ 0.012094] Yama: becoming mindful. [ 0.013048] SELinux: Initializing. [ 0.014080] *** VALIDATE selinux *** [ 0.022360] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027589] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028210] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030162] *** VALIDATE tmpfs *** [ 0.032172] *** VALIDATE proc *** [ 0.033314] *** VALIDATE cgroup *** [ 0.034014] *** VALIDATE cgroup2 *** [ 0.035318] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038029] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042705] debug: unmapping init [mem 0xffffffffaf859000-0xffffffffaf860fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045794] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046045] ... version: 2 [ 0.047018] ... bit width: 48 [ 0.048020] ... generic registers: 4 [ 0.049017] ... value mask: 0000ffffffffffff [ 0.050021] ... max period: 00007fffffffffff [ 0.051021] ... fixed-purpose events: 3 [ 0.052018] ... event mask: 000000070000000f [ 0.053320] rcu: Hierarchical SRCU implementation. [ 0.055775] smp: Bringing up secondary CPUs ... [ 0.056732] x86: Booting SMP configuration: [ 0.057041] .... node #0, CPUs: #1 #2 #3 [ 0.060665] smp: Brought up 1 node, 4 CPUs [ 0.062018] smpboot: Max logical packages: 1 [ 0.063028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.212043] node 0 deferred pages initialised in 146ms [ 0.215153] devtmpfs: initialized [ 0.216214] x86/mm: Memory block size: 128MB [ 0.218265] gcov: version magic: 0x41383552 [ 0.220298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.223108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.224353] pinctrl core: initialized pinctrl subsystem [ 0.226119] [ 0.226454] ************************************************************* [ 0.228015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.229009] ** ** [ 0.230009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.232010] ** ** [ 0.234015] ** This means that this kernel is built to expose internal ** [ 0.235010] ** IOMMU data structures, which may compromise security on ** [ 0.236010] ** your system. ** [ 0.238009] ** ** [ 0.239007] ** If you see this message and you are not debugging the ** [ 0.240013] ** kernel, report this immediately to your vendor! ** [ 0.242009] ** ** [ 0.243009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.244009] ************************************************************* [ 0.245611] NET: Registered protocol family 16 [ 0.247329] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.249080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.252105] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.254720] cpuidle: using governor menu [ 0.256049] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.257703] PCI: Using configuration type 1 for base access [ 0.258163] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.265104] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.268048] cryptd: max_cpu_qlen set to 1000 [ 0.271213] ACPI: Added _OSI(Module Device) [ 0.272000] ACPI: Added _OSI(Processor Device) [ 0.272000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.273035] ACPI: Added _OSI(Processor Aggregator Device) [ 0.278438] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.286552] ACPI: Interpreter enabled [ 0.288079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.290014] ACPI: Using IOAPIC for interrupt routing [ 0.291143] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.295564] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.307015] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.310067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.312028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.316117] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.321668] acpiphp: Slot [2] registered [ 0.323295] acpiphp: Slot [5] registered [ 0.325241] acpiphp: Slot [6] registered [ 0.327210] acpiphp: Slot [7] registered [ 0.328308] acpiphp: Slot [8] registered [ 0.330208] acpiphp: Slot [9] registered [ 0.332222] acpiphp: Slot [10] registered [ 0.334134] acpiphp: Slot [3] registered [ 0.336136] acpiphp: Slot [4] registered [ 0.337128] acpiphp: Slot [11] registered [ 0.339125] acpiphp: Slot [12] registered [ 0.340112] acpiphp: Slot [13] registered [ 0.342140] acpiphp: Slot [14] registered [ 0.343203] acpiphp: Slot [15] registered [ 0.345094] acpiphp: Slot [16] registered [ 0.346107] acpiphp: Slot [17] registered [ 0.347118] acpiphp: Slot [18] registered [ 0.349105] acpiphp: Slot [19] registered [ 0.350141] acpiphp: Slot [20] registered [ 0.351087] acpiphp: Slot [21] registered [ 0.352088] acpiphp: Slot [22] registered [ 0.354106] acpiphp: Slot [23] registered [ 0.355104] acpiphp: Slot [24] registered [ 0.357118] acpiphp: Slot [25] registered [ 0.358110] acpiphp: Slot [26] registered [ 0.360138] acpiphp: Slot [27] registered [ 0.361134] acpiphp: Slot [28] registered [ 0.362100] acpiphp: Slot [29] registered [ 0.364111] acpiphp: Slot [30] registered [ 0.365114] acpiphp: Slot [31] registered [ 0.366098] PCI host bridge to bus 0000:00 [ 0.368022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.370025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.372029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.375031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.378029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.380025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.381134] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.382745] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.384890] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.392015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.396042] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.400022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.401011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.402472] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.404752] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.407050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.410573] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.416000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.426000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.432036] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.438000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.444018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.451016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.464016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.475147] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.481023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.487025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.502029] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.511062] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.514017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.517025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.524018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.530268] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.538021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.546023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.568075] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.584300] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.591024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.597026] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.625022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.635053] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.645030] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.657025] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.672022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.682891] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.685552] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.690527] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.692487] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.695294] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.701073] iommu: Default domain type: Passthrough [ 0.702491] SCSI subsystem initialized [ 0.704127] ACPI: bus type USB registered [ 0.705084] usbcore: registered new interface driver usbfs [ 0.706096] usbcore: registered new interface driver hub [ 0.708078] usbcore: registered new device driver usb [ 0.709171] pps_core: LinuxPPS API ver. 1 registered [ 0.710019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.713069] PTP clock support registered [ 0.714140] EDAC MC: Ver: 3.0.0 [ 0.716109] PCI: Using ACPI for IRQ routing [ 0.717747] NetLabel: Initializing [ 0.719017] NetLabel: domain hash size = 128 [ 0.720011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.722093] NetLabel: unlabeled traffic allowed by default [ 0.724135] vgaarb: loaded [ 0.725296] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.727018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.734021] clocksource: Switched to clocksource kvm-clock [ 0.833717] VFS: Disk quotas dquot_6.6.0 [ 0.835108] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.837025] *** VALIDATE ramfs *** [ 0.837962] *** VALIDATE hugetlbfs *** [ 0.839418] pnp: PnP ACPI init [ 0.841682] pnp: PnP ACPI: found 6 devices [ 0.863287] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.865842] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.867636] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.869306] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.871169] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.872966] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.875042] NET: Registered protocol family 2 [ 0.877117] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.880954] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.883974] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.888280] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.891056] TCP: Hash tables configured (established 65536 bind 65536) [ 0.893603] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.896338] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.898460] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.901213] NET: Registered protocol family 1 [ 0.903868] RPC: Registered named UNIX socket transport module. [ 0.905712] RPC: Registered udp transport module. [ 0.908057] RPC: Registered tcp transport module. [ 0.909572] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.911435] NET: Registered protocol family 44 [ 0.912773] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.914751] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.916813] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.918813] PCI: CLS 0 bytes, default 64 [ 0.920184] Unpacking initramfs... [ 2.363920] debug: unmapping init [mem 0xffff94b2fcc54000-0xffff94b2fffbffff] [ 2.367926] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.369650] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.372169] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.872727] Initialise system trusted keyrings [ 2.874353] Key type blacklist registered [ 2.876300] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.885509] zbud: loaded [ 2.888824] *** VALIDATE nfs *** [ 2.890202] *** VALIDATE nfs4 *** [ 2.891845] pstore: using deflate compression [ 2.895627] Platform Keyring initialized [ 3.001832] NET: Registered protocol family 38 [ 3.003630] Key type asymmetric registered [ 3.005103] Asymmetric key parser 'x509' registered [ 3.006972] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.010157] io scheduler mq-deadline registered [ 3.011736] io scheduler kyber registered [ 3.013365] io scheduler bfq registered [ 3.015288] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.018681] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.021504] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.024199] ACPI: Power Button [PWRF] [ 3.030988] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.037514] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.050917] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.059403] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.085990] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.116740] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.147746] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.154727] Non-volatile memory driver v1.3 [ 3.156501] Linux agpgart interface v0.103 [ 3.199748] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.202646] vda: detected capacity change from 0 to 74838016 [ 3.220897] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.223697] vdb: detected capacity change from 0 to 1073741824 [ 3.237780] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.240511] vdc: detected capacity change from 0 to 2621440000 [ 3.255648] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.259088] vdd: detected capacity change from 0 to 2621440000 [ 3.274587] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.277715] vde: detected capacity change from 0 to 4294967296 [ 3.292801] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.295932] vdf: detected capacity change from 0 to 4294967296 [ 3.304867] libphy: Fixed MDIO Bus: probed [ 3.317913] usbcore: registered new interface driver usbserial_generic [ 3.320089] usbserial: USB Serial support registered for generic [ 3.321537] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.325155] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.326900] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.329301] mousedev: PS/2 mouse device common for all mice [ 3.332265] rtc_cmos 00:05: RTC can wake from S4 [ 3.335092] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.336648] rtc_cmos 00:05: registered as rtc0 [ 3.339845] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.341449] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.343023] intel_pstate: CPU model not supported [ 3.347979] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.348856] hid: raw HID events driver (C) Jiri Kosina [ 3.353212] usbcore: registered new interface driver usbhid [ 3.354218] usbhid: USB HID core driver [ 3.355182] drop_monitor: Initializing network drop monitor service [ 3.356502] Initializing XFRM netlink socket [ 3.357764] NET: Registered protocol family 10 [ 3.359707] Segment Routing with IPv6 [ 3.360448] NET: Registered protocol family 17 [ 3.361834] mpls_gso: MPLS GSO support [ 3.371135] RAS: Correctable Errors collector initialized. [ 3.373151] AVX version of gcm_enc/dec engaged. [ 3.374579] AES CTR mode by8 optimization enabled [ 3.447645] sched_clock: Marking stable (3447612483, 0)->(4453278828, -1005666345) [ 3.450577] registered taskstats version 1 [ 3.452403] Loading compiled-in X.509 certificates [ 3.454283] zswap: loaded using pool lzo/zbud [ 3.480661] Key type big_key registered [ 3.493631] Key type encrypted registered [ 3.495158] ima: No TPM chip found, activating TPM-bypass! [ 3.496808] ima: Allocated hash algorithm: sha1 [ 3.498280] ima: No architecture policies found [ 3.499987] evm: Initialising EVM extended attributes: [ 3.501707] evm: security.selinux [ 3.502756] evm: security.ima [ 3.503680] evm: security.capability [ 3.504667] evm: HMAC attrs: 0x1 [ 3.507081] rtc_cmos 00:05: setting system clock to 2026-08-26 07:38:02 UTC (1787729882) [ 3.513638] debug: unmapping init [mem 0xffffffffb0803000-0xffffffffb09fffff] [ 3.516752] debug: unmapping init [mem 0xffffffffaf582000-0xffffffffaf858fff] [ 3.525108] Write protecting the kernel read-only data: 28672k [ 3.528761] debug: unmapping init [mem 0xffffffffadc03000-0xffffffffaddfffff] [ 3.531702] debug: unmapping init [mem 0xffffffffae514000-0xffffffffae5fffff] [ 3.571207] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.581524] systemd[1]: Detected virtualization kvm. [ 3.583728] systemd[1]: Detected architecture x86-64. [ 3.585761] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.612777] systemd[1]: No hostname configured. [ 3.614660] systemd[1]: Set hostname to . [ 3.616570] random: systemd: uninitialized urandom read (16 bytes read) [ 3.619312] systemd[1]: Initializing machine ID from random generator. [ 3.664926] random: ln: uninitialized urandom read (6 bytes read) [ 3.758592] random: systemd: uninitialized urandom read (16 bytes read) [ 3.761728] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.766296] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.770088] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.418595] device-mapper: uevent: version 1.0.3 [ 4.420983] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 4.978625] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.175160] virtio_net virtio0 ens2: renamed from eth0 [ 5.250370] scsi host0: ata_piix [ 5.279184] scsi host1: ata_piix [ 5.281096] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.283454] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.008948] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.009376] random: crng init done [ 10.010784] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.471582] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.637306] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.890187] SELinux: Disabled at runtime. [ 11.952801] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.958624] systemd[1]: Detected virtualization kvm. [ 11.959879] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.434229] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.438163] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.443567] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.447772] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.451216] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.459378] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.463354] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ 12.640890] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.902831] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.234590] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.312607] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.456497] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.469721] EDAC sbridge: Ver: 1.1.2 [ 14.792265] Key type dns_resolver registered [ 15.091322] NFS: Registering the id_resolver key type [ 15.092741] Key type id_resolver registered [ 15.093836] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg149-server login: [ 40.464457] spl: loading out-of-tree module taints kernel. [ 46.035885] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 57.288142] hrtimer: interrupt took 5132449 ns [ 62.714697] Key type ._llcrypt registered [ 62.718661] Key type .llcrypt registered [ 62.882843] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_hostid [ 84.933175] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 86.456868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.477864] alg: No test for adler32 (adler32-zlib) [ 88.134345] Lustre: Lustre: Build Version: 2.17.57_80_g62ff528 [ 88.630471] LNet: Added LNI 192.168.201.149@tcp [8/256/0/180] [ 90.359182] Key type lgssc registered [ 92.170635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 102.521856] vdc: vdc1 vdc9 [ 115.026827] vdd: vdd1 vdd9 [ 131.425829] vde: vde1 vde9 [ 141.197479] vdf: vdf1 vdf9 [ 161.305153] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing load_modules_local [ 170.154942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.418539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.661307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.772378] Lustre: lustre-MDT0000: new disk, initializing [ 172.151275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.196797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 175.687316] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.355711] Lustre: 6442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 186.500952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 186.511529] Lustre: Skipped 1 previous similar message [ 186.607534] Lustre: lustre-MDT0001: new disk, initializing [ 186.958094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.074238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.089903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 192.214584] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.260621] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.409297] Lustre: lustre-OST0000: new disk, initializing [ 204.414837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.612458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 212.301274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 212.321581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 212.903197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 213.795973] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.080572] Lustre: lustre-OST0001: new disk, initializing [ 227.090271] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 227.427781] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.143758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 235.158719] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 235.622508] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 237.339835] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.959447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.858238] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 265.929860] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.679655] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing yml_node [ 276.708797] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 279.180712] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 281.816714] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 283.484198] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Wed Aug 26 03:42:40 EDT 2026 [ 301.784228] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 304.422662] Lustre: DEBUG MARKER: === replay-single: start setup 03:43:01 (1787730181) === [ 310.424764] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing check_config_client /mnt/lustre [ 334.266154] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 338.373663] Lustre: 12897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 342.065846] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 345.614348] Lustre: DEBUG MARKER: === replay-single: finish setup 03:43:43 (1787730223) === [ 347.659315] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 03:43:45 (1787730225) [ 351.370202] LustreError: 13396:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 352.240500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 354.691880] Lustre: Failing over lustre-MDT0000 [ 354.978074] Lustre: server umount lustre-MDT0000 complete [ 355.808804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 355.827764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 358.371593] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.062313] LustreError: 6449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.083162] LustreError: 6449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 363.488553] LustreError: 6453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.508261] LustreError: 6453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 367.182203] LustreError: 8404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.300667] LustreError: 8404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.311738] LustreError: 8404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 373.693882] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 374.179877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.421846] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 378.114404] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 379.371853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 379.483246] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 387.360488] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 388.623552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 398.563917] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 03:44:35 (1787730275) [ 400.895741] Lustre: Failing over lustre-OST0000 [ 400.989323] Lustre: server umount lustre-OST0000 complete [ 402.405361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 402.420292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 402.434478] Lustre: Skipped 2 previous similar messages [ 402.441895] LustreError: 9541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 402.457959] LustreError: 9541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 404.960338] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 413.792278] LustreError: 9541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 413.825696] LustreError: 9541:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 419.597714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.894917] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 421.410114] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 421.413236] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 421.434642] Lustre: Skipped 3 previous similar messages [ 426.778605] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 435.932429] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 437.470301] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 448.329641] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 03:45:25 (1787730325) [ 451.740387] LustreError: 16430:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 452.644749] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 455.192414] Lustre: Failing over lustre-MDT0000 [ 455.438795] Lustre: server umount lustre-MDT0000 complete [ 455.657499] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.677522] LustreError: 6448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.692214] LustreError: 6448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 473.943321] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.718704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 479.510509] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 479.714243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 479.715855] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 479.718992] Lustre: Skipped 1 previous similar message [ 482.021304] Lustre: lustre-MDT0000: Denying connection for new client 45b9ee12-19ff-4dc8-9e02-f9aa816616aa (at 192.168.201.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 487.507354] Lustre: lustre-MDT0000: Denying connection for new client 45b9ee12-19ff-4dc8-9e02-f9aa816616aa (at 192.168.201.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [ 492.625691] Lustre: lustre-MDT0000: Denying connection for new client 45b9ee12-19ff-4dc8-9e02-f9aa816616aa (at 192.168.201.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 497.747458] Lustre: lustre-MDT0000: Denying connection for new client 45b9ee12-19ff-4dc8-9e02-f9aa816616aa (at 192.168.201.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:51 [ 502.859264] Lustre: lustre-MDT0000: Denying connection for new client 45b9ee12-19ff-4dc8-9e02-f9aa816616aa (at 192.168.201.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:46 [ 513.106771] Lustre: lustre-MDT0000: Denying connection for new client 45b9ee12-19ff-4dc8-9e02-f9aa816616aa (at 192.168.201.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:36 [ 513.126849] Lustre: Skipped 1 previous similar message [ 533.581412] Lustre: lustre-MDT0000: Denying connection for new client 45b9ee12-19ff-4dc8-9e02-f9aa816616aa (at 192.168.201.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:15 [ 533.604196] Lustre: Skipped 3 previous similar messages [ 549.500911] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 549.515839] Lustre: 17070:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0e099fe9-afd9-4697-87fb-ed798d8a5caa@ [ 549.528213] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 549.573196] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 549.574696] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 549.604190] Lustre: Skipped 2 previous similar messages [ 549.626616] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 549.631731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 562.427743] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 03:47:20 (1787730440) [ 565.299759] LustreError: 17817:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 566.183786] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 568.050088] Lustre: Failing over lustre-MDT0000 [ 568.322481] Lustre: server umount lustre-MDT0000 complete [ 570.338044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 570.345857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.360369] Lustre: Skipped 3 previous similar messages [ 570.362748] LustreError: 14351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.378801] LustreError: 14351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 586.618095] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 586.906035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.224751] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 591.854453] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 591.862475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 594.192163] Lustre: lustre-MDT0000: Denying connection for new client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 594.201237] Lustre: Skipped 3 previous similar messages [ 661.074179] Lustre: lustre-MDT0000: Denying connection for new client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:00 [ 661.102494] Lustre: Skipped 12 previous similar messages [ 661.500201] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 661.502914] Lustre: 18441:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 45b9ee12-19ff-4dc8-9e02-f9aa816616aa@ [ 661.531284] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 661.612255] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 661.612487] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 661.630075] Lustre: Skipped 2 previous similar messages [ 661.656911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 661.656929] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 674.594411] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 03:49:12 (1787730552) [ 677.625138] LustreError: 19188:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 678.516449] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 680.464096] Lustre: Failing over lustre-MDT0000 [ 680.811319] Lustre: server umount lustre-MDT0000 complete [ 681.593098] LustreError: 8404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.620786] LustreError: 8404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 682.472533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 682.479390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.489242] Lustre: Skipped 3 previous similar messages [ 699.873036] Lustre: 3454:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787730562/real 1787730562] req@ffff94b350d90e00 x1874570738162432/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787730578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 699.915769] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 709.115548] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdafe944e4e795fd1 [ 709.142112] Lustre: MGC192.168.201.149@tcp: Connection restored to 0@lo (at 0@lo) [ 709.743603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 709.899933] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 710.955527] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 713.876448] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 714.873051] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 714.929692] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 714.932105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 723.570311] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 725.010663] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 733.052885] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 03:50:10 (1787730610) [ 735.691368] LustreError: 20782:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 736.524512] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 738.174786] Lustre: Failing over lustre-MDT0000 [ 738.465501] Lustre: server umount lustre-MDT0000 complete [ 740.331530] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 740.351991] Lustre: Skipped 6 previous similar messages [ 755.758903] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 756.415268] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 757.164617] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 760.461979] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 761.325564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 761.343390] Lustre: Skipped 4 previous similar messages [ 761.512143] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 761.546950] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 761.546950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 768.974591] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 770.666373] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 778.780236] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 03:50:56 (1787730656) [ 781.881436] LustreError: 22357:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 782.654668] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 784.731720] Lustre: Failing over lustre-MDT0000 [ 784.986326] Lustre: server umount lustre-MDT0000 complete [ 802.274458] Lustre: 3455:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787730665/real 1787730665] req@ffff94b249e36300 x1874570738225536/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787730681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 802.315821] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 811.598158] LustreError: 14004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 811.617925] LustreError: 14004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 80 previous similar messages [ 812.525790] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdafe944e4e796a19 [ 813.317584] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 814.650306] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 818.418695] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 818.491052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:193) [ 818.504547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 818.896585] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 830.320949] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 832.122364] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 841.279857] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 03:51:58 (1787730718) [ 844.365946] LustreError: 23955:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 845.697655] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 848.007366] Lustre: Failing over lustre-MDT0000 [ 848.438384] Lustre: server umount lustre-MDT0000 complete [ 848.867055] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 848.895040] Lustre: Skipped 6 previous similar messages [ 867.258482] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 867.563425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 867.566829] Lustre: Skipped 2 previous similar messages [ 867.626398] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 870.987793] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 872.946329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 872.951601] Lustre: Skipped 8 previous similar messages [ 873.188458] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 873.254583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 873.254788] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 873.680592] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 883.602973] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 885.536590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 894.557852] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 03:52:52 (1787730772) [ 897.349636] LustreError: 25535:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 898.191717] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 899.827747] Lustre: Failing over lustre-MDT0000 [ 900.035064] Lustre: server umount lustre-MDT0000 complete [ 919.113540] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 919.862569] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 925.021427] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 925.478504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 925.484202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 937.106791] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 939.171696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 947.895310] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 03:53:45 (1787730825) [ 949.085186] Lustre: *** cfs_fail_loc=13b, val=315*** [ 949.094855] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 949.099875] LustreError: 6451:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b248814a80 x1874570718100096/t38654705666(0) o35->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:314/0 lens 392/456 e 0 to 0 dl 1787730839 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 955.685206] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 958.441974] Lustre: Failing over lustre-MDT0000 [ 959.010134] Lustre: server umount lustre-MDT0000 complete [ 977.377517] Lustre: 3457:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787730839/real 1787730839] req@ffff94b375cd7b80 x1874570738330880/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787730855 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 987.618185] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdafe944e4e797923 [ 988.240814] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 989.007086] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 989.010934] Lustre: Skipped 1 previous similar message [ 993.402120] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 993.409309] Lustre: Skipped 1 previous similar message [ 993.466150] Lustre: 6451:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b248975c00 x1874570718100096/t38654705666(0) o35->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:358/0 lens 392/456 e 0 to 0 dl 1787730883 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 993.495375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 993.495447] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 994.318083] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1004.713527] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1006.479330] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1014.901666] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 03:54:52 (1787730892) [ 1018.512659] LustreError: 28739:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1018.518353] LustreError: 28739:0:(osd_handler.c:715:osd_ro()) Skipped 1 previous similar message [ 1019.503873] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1021.652452] Lustre: Failing over lustre-MDT0000 [ 1021.904811] Lustre: server umount lustre-MDT0000 complete [ 1023.978747] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.998293] Lustre: Skipped 11 previous similar messages [ 1039.270244] Lustre: 3456:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787730902/real 1787730902] req@ffff94b37e6d5180 x1874570738367616/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787730918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1039.314892] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1039.338885] LustreError: Skipped 1 previous similar message [ 1049.569276] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b3478b8700 x1874570738374528/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1050.190591] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1055.213154] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1055.217154] Lustre: Skipped 12 previous similar messages [ 1055.479480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1055.482889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1055.515221] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1065.763064] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1067.692225] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1077.172133] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 03:55:54 (1787730954) [ 1081.118477] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1081.925064] Lustre: *** cfs_fail_loc=114, val=0*** [ 1084.316244] Lustre: Failing over lustre-MDT0000 [ 1084.603985] Lustre: server umount lustre-MDT0000 complete [ 1085.920987] LustreError: 6449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1085.946564] LustreError: 6449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 114 previous similar messages [ 1102.313466] Lustre: 3456:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787730964/real 1787730964] req@ffff94b37ddbf480 x1874570738402432/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787730980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1112.579264] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdafe944e4e7982bc [ 1113.400221] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1118.269968] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1118.880149] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1118.881563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1129.722865] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1131.907645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1142.242446] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 03:56:59 (1787731019) [ 1146.318748] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1147.128941] Lustre: *** cfs_fail_loc=128, val=0*** [ 1149.658967] Lustre: Failing over lustre-MDT0000 [ 1149.922478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1149.929287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1149.960345] Lustre: server umount lustre-MDT0000 complete [ 1169.511312] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.517688] LustreError: Skipped 1 previous similar message [ 1170.060882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.064823] Lustre: Skipped 4 previous similar messages [ 1171.259359] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1171.267551] Lustre: Skipped 2 previous similar messages [ 1175.130685] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1175.133508] Lustre: Skipped 2 previous similar messages [ 1175.173472] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1175.174861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1175.190179] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1184.706659] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1186.293439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1195.913161] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 03:57:53 (1787731073) [ 1199.404517] LustreError: 33687:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1199.413912] LustreError: 33687:0:(osd_handler.c:715:osd_ro()) Skipped 2 previous similar messages [ 1200.731627] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1203.114589] Lustre: Failing over lustre-MDT0000 [ 1203.422272] Lustre: server umount lustre-MDT0000 complete [ 1221.087141] Lustre: 3456:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787731084/real 1787731084] req@ffff94b375f90000 x1874570738472064/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787731100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1231.328687] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b376729180 x1874570738479616/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1232.536637] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1232.546309] Lustre: Skipped 1 previous similar message [ 1238.124583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1238.125474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1238.845820] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1250.344974] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1252.266797] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1262.228978] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 03:58:59 (1787731139) [ 1267.391424] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1270.079196] Lustre: Failing over lustre-MDT0000 [ 1270.358604] Lustre: server umount lustre-MDT0000 complete [ 1289.703599] Lustre: 3454:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787731152/real 1787731152] req@ffff94b375beca80 x1874570738511232/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787731168 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1299.945459] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdafe944e4e79998a [ 1305.146592] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1305.975275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1305.975646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1315.178744] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1317.022596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1326.100601] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 04:00:03 (1787731203) [ 1330.494159] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1339.683656] Lustre: Failing over lustre-MDT0000 [ 1340.108142] Lustre: server umount lustre-MDT0000 complete [ 1341.412876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.454139] Lustre: Skipped 18 previous similar messages [ 1357.791479] Lustre: 3456:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787731220/real 1787731220] req@ffff94b375945500 x1874570738551680/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787731236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1367.026913] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdafe944e4e79cec0 [ 1367.057239] Lustre: MGC192.168.201.149@tcp: Connection restored to 0@lo (at 0@lo) [ 1367.066778] Lustre: Skipped 21 previous similar messages [ 1367.757972] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1367.778471] Lustre: Skipped 1 previous similar message [ 1373.802768] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1378.768849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1378.769553] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1387.792859] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1389.967034] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1418.261953] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 04:01:35 (1787731295) [ 1423.358983] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1425.696812] Lustre: Failing over lustre-MDT0000 [ 1426.051069] Lustre: server umount lustre-MDT0000 complete [ 1444.112557] Lustre: 3457:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787731307/real 1787731307] req@ffff94b37defca80 x1874570738653440/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787731323 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1444.132635] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1444.141446] LustreError: Skipped 3 previous similar messages [ 1456.546205] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1456.558097] Lustre: Skipped 3 previous similar messages [ 1460.113297] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1460.330274] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1460.341093] Lustre: Skipped 3 previous similar messages [ 1460.399329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1460.415468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1469.708898] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1471.188459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1481.596675] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 04:02:39 (1787731359) [ 1484.289526] LustreError: 40089:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1484.293739] LustreError: 40089:0:(osd_handler.c:715:osd_ro()) Skipped 3 previous similar messages [ 1485.030439] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1486.859707] Lustre: Failing over lustre-MDT0000 [ 1487.269085] Lustre: server umount lustre-MDT0000 complete [ 1509.558142] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1511.054799] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1511.057261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1519.585941] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1520.931790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1529.887234] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 04:03:26 (1787731406) [ 1534.547272] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1537.083900] Lustre: Failing over lustre-MDT0000 [ 1537.308347] Lustre: server umount lustre-MDT0000 complete [ 1563.455450] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1563.890926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1563.893127] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1574.266515] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1576.430836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1586.516267] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 04:04:23 (1787731463) [ 1591.550463] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1593.645925] Lustre: Failing over lustre-MDT0000 [ 1593.961098] Lustre: server umount lustre-MDT0000 complete [ 1599.460905] LustreError: 14004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1599.481047] LustreError: 14004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 219 previous similar messages [ 1610.191892] Lustre: 3456:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787731473/real 1787731473] req@ffff94b3478b8000 x1874570738750464/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787731489 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1626.573152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1626.574175] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1630.268233] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1642.344598] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1644.263470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1653.230981] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 04:05:30 (1787731530) [ 1658.085627] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1660.173402] Lustre: Failing over lustre-MDT0000 [ 1660.730640] Lustre: server umount lustre-MDT0000 complete [ 1688.967586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.972774] Lustre: Skipped 7 previous similar messages [ 1689.045962] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1689.060426] Lustre: Skipped 4 previous similar messages [ 1694.298930] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1694.388222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1694.389790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1705.611475] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1707.192686] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1718.698970] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 04:06:35 (1787731595) [ 1723.657629] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1726.067139] Lustre: Failing over lustre-MDT0000 [ 1726.303902] Lustre: server umount lustre-MDT0000 complete [ 1750.199665] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1750.658644] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1750.661951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1759.166091] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1760.923785] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1769.202496] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 04:07:26 (1787731646) [ 1772.757142] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1774.814576] Lustre: Failing over lustre-MDT0000 [ 1775.152669] Lustre: server umount lustre-MDT0000 complete [ 1802.720071] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b37650f800 x1874570738865664/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1808.169778] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1808.603117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1808.606678] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1818.182513] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1819.898242] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1828.295819] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 04:08:25 (1787731705) [ 1831.974522] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1833.750828] Lustre: Failing over lustre-MDT0000 [ 1833.959888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1834.030622] Lustre: server umount lustre-MDT0000 complete [ 1855.659187] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1856.657169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1856.657770] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1865.361708] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1866.946912] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1874.906519] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 04:09:12 (1787731752) [ 1878.813983] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1880.642278] Lustre: Failing over lustre-MDT0000 [ 1880.869325] Lustre: server umount lustre-MDT0000 complete [ 1882.084371] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.091789] Lustre: Skipped 34 previous similar messages [ 1900.584469] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1902.574088] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1902.576697] Lustre: Skipped 36 previous similar messages [ 1902.666087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1902.668736] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1907.935183] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1909.410365] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1916.988370] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 04:09:54 (1787731794) [ 1920.196844] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1921.917404] Lustre: Failing over lustre-MDT0000 [ 1922.130185] Lustre: server umount lustre-MDT0000 complete [ 1923.040066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1942.708763] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1944.646231] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 1944.647317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 1951.439176] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1952.926552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1959.984707] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 04:10:37 (1787731837) [ 1963.476840] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1965.046470] Lustre: Failing over lustre-MDT0000 [ 1965.161100] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 1965.163969] Lustre: Skipped 3 previous similar messages [ 1965.276563] Lustre: server umount lustre-MDT0000 complete [ 1970.147221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1982.374693] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1982.378946] LustreError: Skipped 9 previous similar messages [ 1985.611253] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1985.619550] Lustre: Skipped 9 previous similar messages [ 1987.172716] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1988.303627] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1988.321935] Lustre: Skipped 9 previous similar messages [ 1988.361826] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1988.364847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1996.594802] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1998.257820] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2005.339625] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 04:11:23 (1787731883) [ 2007.456657] LustreError: 55912:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2007.462677] LustreError: 55912:0:(osd_handler.c:715:osd_ro()) Skipped 9 previous similar messages [ 2008.136587] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2009.729564] Lustre: Failing over lustre-MDT0000 [ 2009.940515] Lustre: server umount lustre-MDT0000 complete [ 2026.576431] Lustre: MGS: Not available for connect from 192.168.201.49@tcp (not set up) [ 2030.852535] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2032.281900] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 2032.287058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 2038.712874] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2040.083499] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2047.036395] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 04:12:04 (1787731924) [ 2050.734282] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2052.622564] Lustre: Failing over lustre-MDT0000 [ 2052.964477] Lustre: server umount lustre-MDT0000 complete [ 2053.090919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2075.982273] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2076.335520] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 2076.335734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:993) [ 2085.034522] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2086.108118] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2093.694841] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 04:12:51 (1787731971) [ 2097.590698] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2099.412322] Lustre: Failing over lustre-MDT0000 [ 2099.583317] Lustre: server umount lustre-MDT0000 complete [ 2117.090503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2117.104582] Lustre: Skipped 3 previous similar messages [ 2122.022621] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2122.887612] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2122.887619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2133.497565] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2135.675688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2145.003949] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 04:13:42 (1787732022) [ 2148.134482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2149.633037] Lustre: Failing over lustre-MDT0000 [ 2149.911906] Lustre: server umount lustre-MDT0000 complete [ 2169.265876] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2171.505040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2171.505523] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2176.013405] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2177.507502] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2185.010044] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 04:14:22 (1787732062) [ 2189.709763] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2192.043223] Lustre: Failing over lustre-MDT0000 [ 2192.354088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2192.364363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2192.415637] Lustre: server umount lustre-MDT0000 complete [ 2200.652662] LustreError: 14004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.687393] LustreError: 14004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 242 previous similar messages [ 2210.327903] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2210.341358] Lustre: Skipped 10 previous similar messages [ 2214.703275] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2215.637215] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 2215.638196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 2223.897278] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2225.379727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2232.784711] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 04:15:10 (1787732110) [ 2237.749523] Lustre: 63832:0:(genops.c:1774:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 651c117a-a2a9-4aa4-8c7f-122b99d35934 at adminstrative request [ 2243.199757] Lustre: Failing over lustre-MDT0000 [ 2243.492451] Lustre: server umount lustre-MDT0000 complete [ 2264.132293] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2265.742703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2265.742747] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 2272.449867] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2273.737174] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2278.244754] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2291.425800] Lustre: DEBUG MARKER: before 4096, after 4096 [ 2297.313913] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 04:16:14 (1787732174) [ 2298.297226] Lustre: 65905:0:(genops.c:1774:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 651c117a-a2a9-4aa4-8c7f-122b99d35934 at adminstrative request [ 2307.710381] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 04:16:25 (1787732185) [ 2311.343858] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2313.094485] Lustre: Failing over lustre-MDT0000 [ 2313.300650] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (stopping) [ 2313.339888] Lustre: server umount lustre-MDT0000 complete [ 2329.825179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2329.827462] Lustre: Skipped 12 previous similar messages [ 2333.212927] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2335.380702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2335.380893] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2341.387304] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2342.853310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2351.392778] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 04:17:09 (1787732229) [ 2354.824564] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2356.940561] Lustre: Failing over lustre-MDT0000 [ 2357.257977] Lustre: server umount lustre-MDT0000 complete [ 2379.294154] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2380.979181] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2381.008169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2387.955781] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2389.755991] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2396.862833] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 04:17:54 (1787732274) [ 2400.397481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2402.153917] Lustre: Failing over lustre-MDT0000 [ 2402.420492] Lustre: server umount lustre-MDT0000 complete [ 2423.510637] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2424.968987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2424.975303] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2432.397274] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2434.040627] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2441.864931] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 04:18:39 (1787732319) [ 2446.129497] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2447.881454] Lustre: Failing over lustre-MDT0000 [ 2448.104478] Lustre: server umount lustre-MDT0000 complete [ 2469.966632] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2471.007488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2471.008236] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2478.620515] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2479.971372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2487.021294] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 04:19:24 (1787732364) [ 2491.172911] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2493.058265] Lustre: Failing over lustre-MDT0000 [ 2493.238868] Lustre: server umount lustre-MDT0000 complete [ 2496.480209] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.483744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2496.501291] Lustre: Skipped 51 previous similar messages [ 2514.446166] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2516.456437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2516.467857] Lustre: Skipped 51 previous similar messages [ 2516.616726] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 2516.625016] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 2522.004396] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2523.730768] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2532.745731] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 04:20:10 (1787732410) [ 2536.578652] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2538.684452] Lustre: Failing over lustre-MDT0000 [ 2538.932549] Lustre: server umount lustre-MDT0000 complete [ 2555.986787] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (not set up) [ 2560.586917] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2561.735628] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2561.739374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2569.626644] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2571.362565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2579.190888] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 04:20:56 (1787732456) [ 2583.249248] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2585.556445] Lustre: Failing over lustre-MDT0000 [ 2585.792121] Lustre: server umount lustre-MDT0000 complete [ 2603.489946] Lustre: 3456:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787732466/real 1787732466] req@ffff94b37619c000 x1874570739374336/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787732482 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2603.559126] Lustre: 3456:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2603.572179] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.582778] LustreError: Skipped 12 previous similar messages [ 2613.729629] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b376636d80 x1874570739383808/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2615.086416] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2615.100626] Lustre: Skipped 12 previous similar messages [ 2617.656660] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2619.527976] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2619.538845] Lustre: Skipped 12 previous similar messages [ 2619.570938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 2619.575088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 2626.078188] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2627.696769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2635.546755] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 04:21:53 (1787732513) [ 2638.500733] LustreError: 77327:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2638.511131] LustreError: 77327:0:(osd_handler.c:715:osd_ro()) Skipped 11 previous similar messages [ 2639.272488] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2641.265324] Lustre: Failing over lustre-MDT0000 [ 2641.612278] Lustre: server umount lustre-MDT0000 complete [ 2665.719168] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 2665.725756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 2665.884490] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2675.796274] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2677.180133] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2685.304423] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 04:22:42 (1787732562) [ 2688.849750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2690.447744] Lustre: Failing over lustre-MDT0000 [ 2690.755619] Lustre: server umount lustre-MDT0000 complete [ 2707.940946] Lustre: 3455:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787732570/real 1787732570] req@ffff94b37ff11f80 x1874570739440000/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787732586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2717.152405] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b367e93480 x1874570739448576/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2722.936605] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 2722.943854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 2723.248990] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2731.795134] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2733.233576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2740.084954] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 04:23:37 (1787732617) [ 2744.030284] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2745.763341] Lustre: Failing over lustre-MDT0000 [ 2746.209919] Lustre: server umount lustre-MDT0000 complete [ 2748.389756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2768.199391] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2770.046724] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 2770.050623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 2777.033393] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2778.640170] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2786.373181] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 04:24:23 (1787732663) [ 2790.285797] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2791.933408] Lustre: Failing over lustre-MDT0000 [ 2792.199836] Lustre: server umount lustre-MDT0000 complete [ 2800.717085] LustreError: 29007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.727096] LustreError: 29007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 230 previous similar messages [ 2812.306475] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2812.323386] Lustre: Skipped 11 previous similar messages [ 2817.136750] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 2817.137460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 2817.149415] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2826.471572] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2827.943230] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2835.809501] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 04:25:13 (1787732713) [ 2836.745494] Lustre: 83540:0:(genops.c:1774:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 651c117a-a2a9-4aa4-8c7f-122b99d35934 at adminstrative request [ 2843.910465] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 04:25:21 (1787732721) [ 2846.087876] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2847.414060] Lustre: Failing over lustre-MDT0000 [ 2847.604142] Lustre: server umount lustre-MDT0000 complete [ 2847.715054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2854.569745] Lustre: lustre-MDT0000: Aborting client recovery [ 2854.572172] LustreError: 84449:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2854.576608] Lustre: 84483:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2854.579756] Lustre: 84483:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 651c117a-a2a9-4aa4-8c7f-122b99d35934@ [ 2854.585137] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2854.618183] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2854.674463] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2854.757423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1479 to 0x280000401:1505) [ 2854.759398] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1480 to 0x2c0000401:1505) [ 2857.899660] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2859.492168] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2875.419774] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 04:25:53 (1787732753) [ 2878.881405] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2880.556892] Lustre: Failing over lustre-MDT0000 [ 2880.773072] Lustre: server umount lustre-MDT0000 complete [ 2888.496411] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2888.525382] Lustre: lustre-MDT0000: Aborting client recovery [ 2888.528297] LustreError: 85955:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2888.533932] Lustre: 85988:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2888.541082] Lustre: 85988:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 2888.553547] Lustre: 85988:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2888.565219] Lustre: 85988:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2888.571183] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2888.619597] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 2888.683861] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 2888.746803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1516 to 0x280000401:1537) [ 2888.750063] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1516 to 0x2c0000401:1537) [ 2892.130108] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2893.805716] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2905.013335] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2905.015164] Lustre: Skipped 1 previous similar message [ 2911.601152] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 04:26:29 (1787732789) [ 2914.833649] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2916.515582] Lustre: Failing over lustre-MDT0000 [ 2916.747977] Lustre: server umount lustre-MDT0000 complete [ 2919.392234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2919.408471] LustreError: Skipped 1 previous similar message [ 2924.890284] Lustre: lustre-MDT0000: Aborting client recovery [ 2924.892691] LustreError: 87467:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2924.897752] Lustre: 87500:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2924.902127] Lustre: 87500:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 2924.906257] Lustre: 87500:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2924.912643] Lustre: 87500:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2924.917326] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2924.953441] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 2925.073422] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 2925.185916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 2925.186088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 2930.154483] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2930.631497] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2951.259914] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 04:27:08 (1787732828) [ 2952.135437] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2952.137418] LustreError: 14004:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b37ea45f80 x1874570719151488/t201863462916(0) o36->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:52/0 lens 512/456 e 0 to 0 dl 1787732842 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 2956.068527] Lustre: Failing over lustre-MDT0000 [ 2956.357138] Lustre: server umount lustre-MDT0000 complete [ 2967.803084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2967.806987] Lustre: Skipped 13 previous similar messages [ 2967.905832] Lustre: lustre-MDT0000: Aborting client recovery [ 2967.912454] LustreError: 88821:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2967.925858] Lustre: 88856:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2967.937503] Lustre: 88856:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 2967.950867] Lustre: 88856:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2967.967609] Lustre: 88856:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2967.974871] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2968.052445] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 2968.133940] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 2968.276317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 2968.276895] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1601) [ 2972.652451] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2973.734633] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2992.954985] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 2994.482485] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 04:27:52 (1787732872) [ 2997.656605] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2999.727135] Lustre: Failing over lustre-MDT0000 [ 2999.933197] Lustre: server umount lustre-MDT0000 complete [ 3007.839268] Lustre: lustre-MDT0000: Aborting client recovery [ 3007.841448] LustreError: 90417:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3007.856242] Lustre: 90450:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3007.863509] Lustre: 90450:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 3007.869904] Lustre: 90450:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3007.880801] Lustre: 90450:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3007.888156] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3007.980538] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3008.056360] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3008.122760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 3008.128238] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1633) [ 3011.616144] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3013.097711] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3030.583264] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 04:28:28 (1787732908) [ 3056.684431] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3058.262815] Lustre: Failing over lustre-MDT0000 [ 3058.452239] Lustre: server umount lustre-MDT0000 complete [ 3059.173665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3059.193138] LustreError: Skipped 2 previous similar messages [ 3078.805704] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3080.776670] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3080.786961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3087.218840] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3088.667631] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3109.520950] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 04:29:47 (1787732987) [ 3130.527866] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3141.573492] Lustre: Failing over lustre-MDT0000 [ 3141.958469] Lustre: server umount lustre-MDT0000 complete [ 3142.111572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.128396] Lustre: Skipped 49 previous similar messages [ 3158.495340] Lustre: 3455:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787733021/real 1787733021] req@ffff94b2494ac380 x1874570739924736/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787733037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3168.736852] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b365969500 x1874570739933184/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3173.458956] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3174.905070] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3174.918668] Lustre: Skipped 51 previous similar messages [ 3177.717853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3177.719302] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3182.132590] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3183.609895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3203.339616] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 04:31:20 (1787733080) [ 3205.183143] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3206.029036] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3206.032953] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3212.200333] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 04:31:29 (1787733089) [ 3232.502354] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3242.522731] Lustre: Failing over lustre-OST0000 [ 3242.594981] Lustre: server umount lustre-OST0000 complete [ 3260.448958] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3260.453336] Lustre: Skipped 6 previous similar messages [ 3261.910745] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3261.917708] Lustre: Skipped 6 previous similar messages [ 3264.950179] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3323.071142] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 04:33:20 (1787733200) [ 3325.905924] LustreError: 97819:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3325.909601] LustreError: 97819:0:(osd_handler.c:715:osd_ro()) Skipped 10 previous similar messages [ 3326.516850] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3328.563752] Lustre: Failing over lustre-MDT0000 [ 3328.756035] Lustre: server umount lustre-MDT0000 complete [ 3345.888318] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3345.895666] LustreError: Skipped 11 previous similar messages [ 3351.016989] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3351.696326] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3351.696908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3360.298646] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3362.322566] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3367.903179] Lustre: 98477:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787733230/real 1787733230] req@ffff94b3765eea00 x1874570740257152/t0(0) o5->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1787733246 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-0-0.0' uid:0 gid:0 projid:4294967295 [ 3367.918392] LustreError: 98477:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3367.923596] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3368.929128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3380.822731] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 04:34:18 (1787733258) [ 3385.729110] LustreError: 6447:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3390.943089] LustreError: 6447:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3390.949075] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnecting [ 3391.001429] LustreError: 38146:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 waking [ 3392.613719] LustreError: 6448:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3398.111172] LustreError: 6448:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3398.115599] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnecting [ 3399.977839] LustreError: 14004:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3405.279175] LustreError: 14004:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3405.289316] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnecting [ 3407.082035] LustreError: 6449:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3412.448879] LustreError: 6449:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3414.871956] LustreError: 29007:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3420.128870] LustreError: 29007:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3420.144356] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnecting [ 3420.155542] Lustre: Skipped 1 previous similar message [ 3429.280705] LustreError: 6449:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3429.288178] LustreError: 6449:0:(ldlm_lib.c:1176:target_handle_connect()) Skipped 1 previous similar message [ 3434.463145] LustreError: 6449:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3434.467755] LustreError: 6449:0:(ldlm_lib.c:1176:target_handle_connect()) Skipped 1 previous similar message [ 3442.143256] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnecting [ 3442.147687] Lustre: Skipped 2 previous similar messages [ 3451.836884] LustreError: 14004:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_race id 701 sleeping [ 3451.842290] LustreError: 14004:0:(ldlm_lib.c:1176:target_handle_connect()) Skipped 2 previous similar messages [ 3456.996109] LustreError: 14004:0:(ldlm_lib.c:1176:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3457.003241] LustreError: 14004:0:(ldlm_lib.c:1176:target_handle_connect()) Skipped 2 previous similar messages [ 3465.084753] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 04:35:42 (1787733342) [ 3467.248774] LustreError: 6447:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3477.579352] Lustre: lustre-MDT0000: Export ffff94b375cdf800 already connecting from 192.168.201.49@tcp [ 3480.657713] Lustre: lustre-MDT0000: Export ffff94b375cdf800 already connecting from 192.168.201.49@tcp [ 3482.703427] Lustre: lustre-MDT0000: Export ffff94b375cdf800 already connecting from 192.168.201.49@tcp [ 3487.819791] Lustre: lustre-MDT0000: Export ffff94b375cdf800 already connecting from 192.168.201.49@tcp [ 3492.940553] Lustre: lustre-MDT0000: Export ffff94b375cdf800 already connecting from 192.168.201.49@tcp [ 3492.950145] Lustre: Skipped 1 previous similar message [ 3503.183734] Lustre: lustre-MDT0000: Export ffff94b375cdf800 already connecting from 192.168.201.49@tcp [ 3503.189596] Lustre: Skipped 1 previous similar message [ 3507.343132] LustreError: 6447:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3507.357066] Lustre: 6447:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94b375f95c00 x1874570721808128/t0(0) o38->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1787733366 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3508.316748] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnecting [ 3508.328032] Lustre: Skipped 3 previous similar messages [ 3508.334819] LustreError: 14004:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3533.902980] Lustre: lustre-MDT0000: Export ffff94b375cdf800 already connecting from 192.168.201.49@tcp [ 3548.383375] LustreError: 14004:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3548.391637] Lustre: 14004:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94b367fc7480 x1874570721812736/t0(0) o38->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1787733407 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3549.260925] LustreError: 6448:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3574.861519] Lustre: lustre-MDT0000: Export ffff94b375cdf800 already connecting from 192.168.201.49@tcp [ 3574.864099] Lustre: Skipped 3 previous similar messages [ 3589.311724] LustreError: 6448:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3589.322426] Lustre: 6448:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94b36bc36300 x1874570721815936/t0(0) o38->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1787733448 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3590.219755] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnecting [ 3590.234140] Lustre: Skipped 1 previous similar message [ 3590.236432] LustreError: 29007:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3630.295151] LustreError: 29007:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3630.299665] Lustre: 29007:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94b367fa6a00 x1874570721819136/t0(0) o38->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1787733489 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3631.180967] LustreError: 6449:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3656.785869] Lustre: lustre-MDT0000: Export ffff94b375cdf800 already connecting from 192.168.201.49@tcp [ 3656.793940] Lustre: Skipped 7 previous similar messages [ 3671.263122] LustreError: 6449:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3671.271151] Lustre: 6449:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff94b365882680 x1874570721822336/t0(0) o38->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:0/0 lens 520/416 e 0 to 0 dl 1787733530 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3672.143385] LustreError: 6448:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3690.528308] LustreError: 6448:0:(ldlm_lib.c:1434:target_handle_connect()) cfs_fail_timeout interrupted [ 3694.794541] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 04:39:32 (1787733572) [ 3699.143725] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3704.543819] Lustre: Failing over lustre-MDT0000 [ 3704.965228] Lustre: server umount lustre-MDT0000 complete [ 3705.311594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3705.323740] LustreError: Skipped 2 previous similar messages [ 3705.337203] LustreError: 100175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3705.354531] LustreError: 100175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 129 previous similar messages [ 3716.403591] Lustre: *** cfs_fail_loc=712, val=0*** [ 3716.406778] LustreError: 16201:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff94b350d90e00 x1874570740424192/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 3716.632313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3716.640258] Lustre: Skipped 5 previous similar messages [ 3716.712245] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3716.714595] Lustre: Skipped 19 previous similar messages [ 3716.715465] Lustre: lustre-MDT0000: Aborting client recovery [ 3716.720587] LustreError: 102988:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3716.724117] Lustre: 103022:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3716.727800] Lustre: 103022:0:(ldlm_lib.c:2429:target_recovery_overseer()) Skipped 2 previous similar messages [ 3716.731285] Lustre: 103022:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3716.735779] Lustre: 103022:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3716.739159] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3716.770537] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 3716.865505] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3716.971769] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 3716.975368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 3720.972630] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3721.701416] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3735.935268] Lustre: Failing over lustre-MDT0000 [ 3736.189392] Lustre: server umount lustre-MDT0000 complete [ 3753.443622] Lustre: 3455:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787733616/real 1787733616] req@ffff94b367528a80 x1874570740441344/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787733632 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3768.363116] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3769.519406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 3769.528369] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 3778.444310] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3780.173712] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3788.820535] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 04:41:06 (1787733666) [ 3788.927348] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnecting [ 3788.936955] Lustre: Skipped 2 previous similar messages [ 3795.882200] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 04:41:13 (1787733673) [ 3796.842240] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3796.848167] LustreError: 84994:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b381e2fb80 x1874570721886208/t0(0) o700->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:141/0 lens 264/248 e 0 to 0 dl 1787733686 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 3816.221847] Lustre: Failing over lustre-MDT0000 [ 3816.659480] Lustre: server umount lustre-MDT0000 complete [ 3820.512496] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.529626] Lustre: Skipped 21 previous similar messages [ 3838.902567] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3839.981671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3839.987251] Lustre: Skipped 19 previous similar messages [ 3840.103047] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 3840.103071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 3848.494336] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3850.005304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3859.122747] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 04:42:17 (1787733737) [ 3861.374361] Lustre: Failing over lustre-OST0000 [ 3861.499575] Lustre: server umount lustre-OST0000 complete [ 3879.133627] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3879.139131] Lustre: Skipped 3 previous similar messages [ 3879.457429] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3879.466512] Lustre: Skipped 3 previous similar messages [ 3882.736380] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3891.075595] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3892.325920] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3964.500298] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 04:44:01 (1787733841) [ 3967.462368] LustreError: 108320:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 3967.476905] LustreError: 108320:0:(osd_handler.c:715:osd_ro()) Skipped 1 previous similar message [ 3968.228506] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3970.511815] Lustre: Failing over lustre-MDT0000 [ 3970.796957] Lustre: server umount lustre-MDT0000 complete [ 3988.945856] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3988.960603] LustreError: Skipped 3 previous similar messages [ 3995.400738] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3995.416566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 3995.417428] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4062.289420] Lustre: *** cfs_fail_loc=216, val=0*** [ 4068.474857] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 04:45:45 (1787733945) [ 4070.722312] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4070.730345] Lustre: Skipped 2 previous similar messages [ 4084.020217] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 04:46:01 (1787733961) [ 4086.857278] Lustre: Failing over lustre-MDT0000 [ 4087.213690] Lustre: server umount lustre-MDT0000 complete [ 4104.795433] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.49@tcp (not set up) [ 4109.516056] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4110.323914] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4110.327660] LustreError: 110549:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b376609c00 x1874570722013312/t0(0) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:495/0 lens 328/344 e 0 to 0 dl 1787734040 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4166.218837] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnected, waiting for 2 clients in recovery for 1:06 [ 4166.240565] Lustre: 110549:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4166.400895] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 4166.401419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 4171.463776] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4173.195851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4182.531637] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 04:47:39 (1787734059) [ 4184.486738] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4188.699808] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4190.838979] Lustre: Failing over lustre-MDT0000 [ 4191.156960] Lustre: server umount lustre-MDT0000 complete [ 4223.502712] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4224.681259] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 4224.683168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 4232.407721] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4234.096349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4241.948602] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 04:48:39 (1787734119) [ 4243.154660] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4248.087971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4249.680944] Lustre: Failing over lustre-MDT0000 [ 4249.936287] Lustre: server umount lustre-MDT0000 complete [ 4266.465760] Lustre: 3454:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787734129/real 1787734129] req@ffff94b37ea59880 x1874570740722304/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787734145 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4266.507957] Lustre: 3454:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4282.554921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3169) [ 4282.554952] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 4282.643096] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4293.358326] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4295.232632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4304.588359] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 04:49:42 (1787734182) [ 4306.018089] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4312.134222] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4314.411608] Lustre: Failing over lustre-MDT0000 [ 4314.642780] Lustre: server umount lustre-MDT0000 complete [ 4318.179132] LustreError: 6452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4318.202196] LustreError: 6452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 176 previous similar messages [ 4333.318985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4333.323721] Lustre: Skipped 7 previous similar messages [ 4333.429818] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4333.435589] Lustre: Skipped 9 previous similar messages [ 4338.981232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3201) [ 4338.996149] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 4339.637931] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4353.607727] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 04:50:31 (1787734231) [ 4355.957846] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4355.960651] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4355.970100] LustreError: 36915:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b34487c450 x1874570722073088/t257698037777(0) o35->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:700/0 lens 392/456 e 0 to 0 dl 1787734245 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4358.615890] Lustre: Failing over lustre-MDT0000 [ 4358.984377] Lustre: server umount lustre-MDT0000 complete [ 4391.337720] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4391.598428] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 4391.601885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3233) [ 4391.605433] Lustre: 6450:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b37e45ce00 x1874570722073088/t257698037777(0) o35->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:736/0 lens 392/456 e 0 to 0 dl 1787734281 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4400.955438] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4402.421970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4411.329570] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 04:51:28 (1787734288) [ 4412.624908] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4412.629169] LustreError: 6448:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b3512f9c00 x1874570722089472/t261993005072(0) o36->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:46/0 lens 504/448 e 0 to 0 dl 1787734346 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4419.711609] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4421.606756] Lustre: Failing over lustre-MDT0000 [ 4421.922977] Lustre: server umount lustre-MDT0000 complete [ 4422.141755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4422.163130] Lustre: Skipped 31 previous similar messages [ 4449.249677] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b3658b1880 x1874570740833792/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4455.429536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4455.438509] Lustre: Skipped 31 previous similar messages [ 4455.511316] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4455.669982] Lustre: 6449:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b248975180 x1874570722089472/t261993005072(0) o36->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:89/0 lens 504/2880 e 0 to 0 dl 1787734389 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4455.676604] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233) [ 4455.688814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 4465.354712] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4466.882439] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4475.821586] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 04:52:33 (1787734353) [ 4477.275223] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4477.286429] LustreError: 6448:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b36bd99180 x1874570722106368/t266287972368(0) o36->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:111/0 lens 504/448 e 0 to 0 dl 1787734411 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4479.657359] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4483.806504] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4485.719990] Lustre: Failing over lustre-MDT0000 [ 4485.906686] Lustre: server umount lustre-MDT0000 complete [ 4515.303777] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4515.307228] Lustre: Skipped 7 previous similar messages [ 4517.571985] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4518.454472] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4518.461513] Lustre: Skipped 7 previous similar messages [ 4518.512690] Lustre: 6448:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b365883800 x1874570722106368/t266287972368(0) o36->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:152/0 lens 504/2880 e 0 to 0 dl 1787734452 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4518.535792] Lustre: 6448:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4518.539295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297) [ 4518.539396] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 4529.775233] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 04:53:27 (1787734407) [ 4530.936953] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4530.939847] Lustre: Skipped 1 previous similar message [ 4530.943680] LustreError: 29007:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b248974700 x1874570722120704/t270582939664(0) o36->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:164/0 lens 504/448 e 0 to 0 dl 1787734464 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4530.966919] LustreError: 29007:0:(ldlm_lib.c:3373:target_send_reply_msg()) Skipped 1 previous similar message [ 4533.022132] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4533.023643] Lustre: Skipped 1 previous similar message [ 4537.522861] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4539.168850] Lustre: Failing over lustre-MDT0000 [ 4539.355165] Lustre: server umount lustre-MDT0000 complete [ 4561.455187] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4562.582163] Lustre: 29007:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b377fb0e00 x1874570722120704/t270582939664(0) o36->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:196/0 lens 504/2880 e 0 to 0 dl 1787734496 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4562.587383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 4562.587582] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3297) [ 4571.057399] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 04:54:08 (1787734448) [ 4572.408787] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4574.417975] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4574.424846] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4574.435749] LustreError: 6450:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b375985500 x1874570722133760/t274877906960(0) o35->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:164/0 lens 392/456 e 0 to 0 dl 1787734464 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4578.193898] LustreError: 122678:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 4578.199526] LustreError: 122678:0:(osd_handler.c:715:osd_ro()) Skipped 6 previous similar messages [ 4578.899203] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4580.601359] Lustre: Failing over lustre-MDT0000 [ 4580.858463] Lustre: server umount lustre-MDT0000 complete [ 4598.186526] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4598.191139] LustreError: Skipped 8 previous similar messages [ 4602.999478] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4604.021117] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3329) [ 4604.023042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 4604.031870] Lustre: 36915:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b3670b4a80 x1874570722133760/t274877906960(0) o35->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:193/0 lens 392/456 e 0 to 0 dl 1787734493 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4612.338421] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 04:54:50 (1787734490) [ 4613.243516] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 4613.257287] LustreError: 6447:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b375945500 x1874570722144896/t279172874255(0) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:247/0 lens 664/608 e 0 to 0 dl 1787734547 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4673.098914] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnecting [ 4673.112489] Lustre: Skipped 1 previous similar message [ 4673.120785] Lustre: 6449:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b365968000 x1874570722144896/t279172874255(0) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:307/0 lens 664/3488 e 0 to 0 dl 1787734607 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 4679.932060] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 04:55:57 (1787734557) [ 4684.225387] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4686.230961] Lustre: Failing over lustre-MDT0000 [ 4686.307008] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4686.315369] LustreError: Skipped 2 previous similar messages [ 4686.337423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4686.433669] Lustre: server umount lustre-MDT0000 complete [ 4708.490282] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4710.014878] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3361) [ 4710.016409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 4718.259703] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4720.198936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4738.838209] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 04:56:56 (1787734616) [ 4745.281559] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4747.485573] Lustre: Failing over lustre-MDT0000 [ 4747.829082] Lustre: server umount lustre-MDT0000 complete [ 4776.418940] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b37fe09180 x1874570741022208/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4782.059570] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4782.334783] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 4782.335190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3425) [ 4791.887472] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4793.312967] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4798.689525] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4809.695589] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 04:58:07 (1787734687) [ 4848.502928] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4850.361350] Lustre: Failing over lustre-MDT0000 [ 4850.879552] Lustre: server umount lustre-MDT0000 complete [ 4869.091335] Lustre: 3454:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787734732/real 1787734732] req@ffff94b3769aea00 x1874570741074944/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787734748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4869.118714] Lustre: 3454:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4879.333757] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdafe944e4e80eca3 [ 4884.623541] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4885.052457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 4885.052536] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 4894.390927] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4895.958798] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4946.766689] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 05:00:23 (1787734823) [ 4951.230107] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4953.655489] Lustre: Failing over lustre-MDT0000 [ 4953.915810] Lustre: server umount lustre-MDT0000 complete [ 4956.641838] LustreError: 8404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4956.655837] LustreError: 8404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 228 previous similar messages [ 4972.682121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4972.685056] Lustre: Skipped 8 previous similar messages [ 4972.737963] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4972.743348] Lustre: Skipped 8 previous similar messages [ 4976.045144] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4977.857853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 4977.860403] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 4985.586618] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4987.110920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4996.925375] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 4998.289607] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5005.485191] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 05:01:22 (1787734882) [ 5012.329909] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5074.480929] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5074.483810] Lustre: Skipped 1 previous similar message [ 5074.492095] LustreError: 6449:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b36bd99c00 x1874570724950400/t296352743435(0) o36->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:708/0 lens 66040/440 e 0 to 0 dl 1787735008 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5134.966367] Lustre: 8404:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b344845850 x1874570724950400/t296352743435(0) o36->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:13/0 lens 66040/440 e 0 to 0 dl 1787735068 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5142.231239] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5143.499229] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 05:03:41 (1787735021) [ 5153.182636] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5155.773412] Lustre: Failing over lustre-MDT0000 [ 5156.039511] Lustre: server umount lustre-MDT0000 complete [ 5156.835658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.843286] Lustre: Skipped 31 previous similar messages [ 5183.909970] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5183.923378] Lustre: Skipped 6 previous similar messages [ 5187.469266] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5188.069974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5188.081933] Lustre: Skipped 32 previous similar messages [ 5189.320122] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 5189.338884] Lustre: Skipped 6 previous similar messages [ 5189.406724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 5189.413393] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 5196.366476] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5197.726972] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5207.449966] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 05:04:45 (1787735085) [ 5222.366963] LustreError: 134357:0:(osd_handler.c:715:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 5222.378148] LustreError: 134357:0:(osd_handler.c:715:osd_ro()) Skipped 5 previous similar messages [ 5223.174254] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5235.012503] Lustre: Failing over lustre-OST0000 [ 5235.091351] Lustre: server umount lustre-OST0000 complete [ 5257.275533] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5271.750792] Lustre: Failing over lustre-OST0000 [ 5271.862447] Lustre: server umount lustre-OST0000 complete [ 5295.732214] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5304.022884] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5305.533485] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5345.503749] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 05:07:03 (1787735223) [ 5348.272491] Lustre: Failing over lustre-MDT0000 [ 5348.335556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5348.344595] LustreError: Skipped 3 previous similar messages [ 5348.351576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5348.504649] Lustre: server umount lustre-MDT0000 complete [ 5365.493970] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5365.501800] LustreError: Skipped 5 previous similar messages [ 5370.225780] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5371.522640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 5371.526744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 5384.350971] Lustre: Failing over lustre-MDT0000 [ 5384.694639] Lustre: server umount lustre-MDT0000 complete [ 5402.081520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 5406.469897] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5407.919905] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 5407.919904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 5415.640542] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5417.253314] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5425.240464] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 05:08:23 (1787735303) [ 5439.416691] Lustre: Failing over lustre-OST0000 [ 5439.527471] Lustre: server umount lustre-OST0000 complete [ 5462.963395] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5471.218566] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5472.716685] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5481.088771] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 05:09:19 (1787735359) [ 5482.415593] Lustre: Failing over lustre-MDT0000 [ 5482.679276] Lustre: server umount lustre-MDT0000 complete [ 5490.259722] Lustre: *** cfs_fail_loc=605, val=0*** [ 5490.261275] LustreError: 141751:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc129dfe0 failed: rc = -95 [ 5490.264960] LustreError: 141751:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 5490.271036] LustreError: 141751:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 5490.273902] LustreError: 141751:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 5490.276350] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5490.279236] LustreError: 141751:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5490.319467] Lustre: server umount lustre-MDT0000 complete [ 5490.320884] LustreError: 141751:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5498.220603] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5500.496601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 5500.499564] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 5504.444368] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 05:09:42 (1787735382) [ 5507.328314] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5509.535416] Lustre: Failing over lustre-MDT0000 [ 5509.794750] Lustre: server umount lustre-MDT0000 complete [ 5525.984182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 5525.986646] Lustre: Skipped 6 previous similar messages [ 5527.143348] Lustre: *** cfs_fail_loc=707, val=0*** [ 5529.173315] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5587.549698] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5587.654960] Lustre: 143471:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5587.663262] Lustre: 143471:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 1 previous similar message [ 5588.179698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 5588.183291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 5594.133033] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5595.674248] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5604.659489] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 05:11:22 (1787735482) [ 5634.607601] LustreError: 6447:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b25152fb80 x1874570725845376/t0(0) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:469/0 lens 664/0 e 0 to 0 dl 1787735524 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5634.630974] LustreError: 6447:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 5645.679126] LustreError: 6447:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5645.695619] LustreError: 36915:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b249a0d180 x1874570725846656/t0(0) o35->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:480/0 lens 392/0 e 0 to 0 dl 1787735535 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5647.401289] LustreError: 6449:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b348024380 x1874570725852544/t0(0) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:521/0 lens 576/0 e 0 to 0 dl 1787735576 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 5647.425580] LustreError: 6449:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 5649.985755] LustreError: 129659:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b35d69d880 x1874570742291456/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:484/0 lens 544/0 e 0 to 0 dl 1787735539 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 5649.997281] LustreError: 129659:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 20 previous similar messages [ 5655.072199] LustreError: 38150:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b347540700 x1874570742294272/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:490/0 lens 544/0 e 0 to 0 dl 1787735545 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 5655.091753] LustreError: 38150:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 7 previous similar messages [ 5662.934172] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 05:12:20 (1787735540) [ 5692.534856] LustreError: 8099:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 5703.575159] LustreError: 8099:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 5711.104601] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 05:13:08 (1787735588) [ 5738.806425] LustreError: 14004:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b3670e3b80 x1874570725869952/t0(0) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:573/0 lens 576/0 e 0 to 0 dl 1787735628 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5738.822146] LustreError: 14004:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 5743.823164] LustreError: 14004:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5756.583152] LustreError: 8404:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5756.603096] LustreError: 29007:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b365968700 x1874570725892096/t0(0) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:630/0 lens 664/0 e 0 to 0 dl 1787735685 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5756.618437] LustreError: 29007:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 120 previous similar messages [ 5773.234215] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 05:14:11 (1787735651) [ 5875.243667] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 05:15:52 (1787735752) [ 5903.874878] LustreError: 6448:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff94b37672a300 x1874570725949952/t0(0) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:738/0 lens 576/0 e 0 to 0 dl 1787735793 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 5903.903278] LustreError: 6448:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 100 previous similar messages [ 5903.913060] LustreError: 6448:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5903.933054] LustreError: 6448:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 5904.359416] LustreError: 6448:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5920.339982] LustreError: 6451:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 5920.354958] LustreError: 6451:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 5920.792516] LustreError: 6451:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 5920.813678] LustreError: 6451:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 5956.500795] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 05:17:13 (1787735833) [ 5990.368383] Lustre: DEBUG MARKER: phase 2 [ 5999.008515] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 05:17:56 (1787735876) [ 6081.071807] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 05:19:18 (1787735958) [ 6082.614810] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6084.304809] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 05:19:21 (1787735961) [ 6089.132612] Lustre: DEBUG MARKER: Started rundbench load pid=131435 ... [ 6092.795621] LustreError: 150282:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 6092.810855] LustreError: 150282:0:(osd_handler.c:715:osd_ro()) Skipped 1 previous similar message [ 6093.591666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6096.283911] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6098.465419] Lustre: Failing over lustre-MDT0000 [ 6098.682456] Lustre: server umount lustre-MDT0000 complete [ 6099.008372] LustreError: 14004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.021992] LustreError: 14004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 137 previous similar messages [ 6099.424106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6099.440051] Lustre: Skipped 26 previous similar messages [ 6115.791558] Lustre: 3456:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787735978/real 1787735978] req@ffff94b376635f80 x1874570742546432/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787735994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6115.821101] Lustre: 3456:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6115.835425] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6115.862882] LustreError: Skipped 3 previous similar messages [ 6125.044823] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdafe944e4e85d570 [ 6125.060103] Lustre: MGC192.168.201.149@tcp: Connection restored to 0@lo (at 0@lo) [ 6125.071314] Lustre: Skipped 25 previous similar messages [ 6125.728282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.733646] Lustre: Skipped 8 previous similar messages [ 6125.851700] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6125.858222] Lustre: Skipped 8 previous similar messages [ 6126.156557] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6126.162290] Lustre: Skipped 7 previous similar messages [ 6129.722852] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6130.792024] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6130.799158] Lustre: Skipped 7 previous similar messages [ 6130.869298] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 6130.870295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 6138.756101] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6139.993470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6147.849867] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6150.503919] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6152.660888] Lustre: Failing over lustre-MDT0001 [ 6152.768774] LustreError: 6447:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000032282fd4 ns: mdt-lustre-MDT0001_UUID lock: ffff94b24a839c00/0xdafe944e4e8639ae lrc: 3/0,0 mode: CW/CW res: [0x2400007ed:0x36b:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.201.49@tcp remote: 0x1bafbbd253cd1fd1 expref: 4 pid: 6447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 6156.264850] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6156.270208] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6156.271803] Lustre: Skipped 1 previous similar message [ 6156.300230] LustreError: Skipped 2 previous similar messages [ 6156.880428] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 6156.887058] Lustre: Skipped 1 previous similar message [ 6159.128935] Lustre: server umount lustre-MDT0001 complete [ 6181.938269] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6183.365331] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:527 to 0x280000400:577) [ 6183.367081] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:509 to 0x2c0000400:545) [ 6190.919086] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6192.280821] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6198.823894] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6201.454285] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 6203.449705] Lustre: Failing over lustre-MDT0000 [ 6203.708800] Lustre: server umount lustre-MDT0000 complete [ 6228.681912] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6229.167563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 6229.174351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 6238.791666] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6241.310155] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6252.759162] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 05:22:09 (1787736129) [ 6377.840125] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6390.018302] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 6391.594496] Lustre: Failing over lustre-MDT0001 [ 6392.802363] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6397.235449] Lustre: server umount lustre-MDT0001 complete [ 6419.899577] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6424.173543] Lustre: 29007:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b37692b100 x1874570730676480/t8589939746(0) o36->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:504/0 lens 488/3152 e 0 to 0 dl 1787736314 ref 1 fl Interpret:/202/0 rc 0/0 job:'tar.0' uid:0 gid:0 projid:4294967295 [ 6424.205770] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:871 to 0x280000400:897) [ 6424.206533] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:840 to 0x2c0000400:865) [ 6429.929744] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6431.811979] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6525.437637] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 05:26:43 (1787736403) [ 6651.268631] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6663.643388] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 6665.722111] Lustre: Failing over lustre-MDT0000 [ 6666.000657] Lustre: server umount lustre-MDT0000 complete [ 6686.869483] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6689.948979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5509 to 0x280000401:5537) [ 6689.952666] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5477 to 0x2c0000401:5505) [ 6695.632619] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6696.957471] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6822.533537] LustreError: 158405:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 6822.542487] LustreError: 158405:0:(osd_handler.c:715:osd_ro()) Skipped 4 previous similar messages [ 6823.565552] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6835.983605] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 6837.754810] Lustre: Failing over lustre-MDT0000 [ 6838.059260] Lustre: server umount lustre-MDT0000 complete [ 6838.752774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6838.759492] LustreError: Skipped 3 previous similar messages [ 6838.764513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6838.783268] Lustre: Skipped 14 previous similar messages [ 6838.790483] LustreError: 129487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.808955] LustreError: 129487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 100 previous similar messages [ 6855.467763] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6855.474420] LustreError: Skipped 2 previous similar messages [ 6855.888806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6855.892985] Lustre: Skipped 4 previous similar messages [ 6855.963542] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6855.970431] Lustre: Skipped 4 previous similar messages [ 6859.602151] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6860.363820] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6860.368670] Lustre: Skipped 4 previous similar messages [ 6861.328724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6861.345100] Lustre: Skipped 18 previous similar messages [ 6863.242751] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6863.259270] Lustre: Skipped 4 previous similar messages [ 6863.317699] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5597 to 0x2c0000401:5633) [ 6863.324092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5630 to 0x280000401:5665) [ 6869.399726] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6870.922437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6878.911774] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 05:32:36 (1787736756) [ 7005.011625] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7017.002308] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 7019.185985] Lustre: Failing over lustre-MDT0000 [ 7019.580036] Lustre: server umount lustre-MDT0000 complete [ 7039.967159] Lustre: 3455:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787736902/real 1787736902] req@ffff94b348590a80 x1874570748773760/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787736918 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7050.208373] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b37e5d0a80 x1874570748782464/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7057.051329] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7060.934759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6105 to 0x280000401:6145) [ 7060.937358] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6073 to 0x2c0000401:6113) [ 7070.153583] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7072.439477] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7084.931179] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 05:36:01 (1787736961) [ 7094.060388] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7097.378395] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7099.421925] Lustre: Failing over lustre-OST0000 [ 7099.506785] Lustre: server umount lustre-OST0000 complete [ 7129.711599] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7141.765852] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7144.100369] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7156.773723] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 05:37:14 (1787737034) [ 7281.651763] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7286.236524] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7298.389987] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 7300.159594] Lustre: Failing over lustre-MDT0001 [ 7300.174413] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7300.176572] Lustre: Skipped 3 previous similar messages [ 7300.178620] LustreError: 6447:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000071000c8f ns: mdt-lustre-MDT0001_UUID lock: ffff94b342b2b800/0xdafe944e4e9852ce lrc: 3/0,0 mode: PR/PR res: [0x24000178b:0x2be:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.201.49@tcp remote: 0x1bafbbd253d10cb3 expref: 3 pid: 6447 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 7300.353578] Lustre: server umount lustre-MDT0001 complete [ 7303.917554] Lustre: Failing over lustre-MDT0000 [ 7304.179609] Lustre: server umount lustre-MDT0000 complete [ 7321.567307] Lustre: 3457:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787737184/real 1787737184] req@ffff94b25654e300 x1874570750500864/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787737200 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7321.600095] Lustre: 3457:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 7323.328540] LustreError: 164992:0:(llog.c:1655:llog_backup()) MGC192.168.201.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7323.338037] Lustre: 164992:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7331.812938] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b252e9b800 x1874570750505088/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7336.517156] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7337.338647] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7338.566549] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1175 to 0x280000400:1217) [ 7338.580298] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1144 to 0x2c0000400:1185) [ 7348.308745] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6155 to 0x2c0000401:6177) [ 7348.318176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6186 to 0x280000401:6209) [ 7354.158680] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7356.041987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7357.828205] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7366.698508] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 05:40:44 (1787737244) [ 7370.736583] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7373.659726] Lustre: Failing over lustre-MDT0000 [ 7373.799643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7373.812056] Lustre: Skipped 4 previous similar messages [ 7373.941801] Lustre: server umount lustre-MDT0000 complete [ 7394.395172] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7398.291318] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7410.804333] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 7410.885144] Lustre: 164998:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b3760c0000 x1874570739593472/t339302416941(339302416941) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:735/0 lens 520/3488 e 0 to 0 dl 1787737300 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7411.066552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6211 to 0x280000401:6241) [ 7411.067122] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6155 to 0x2c0000401:6209) [ 7417.273431] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7418.692396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7426.806223] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 05:41:44 (1787737304) [ 7429.871069] LustreError: 168100:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 7429.881092] LustreError: 168100:0:(osd_handler.c:715:osd_ro()) Skipped 5 previous similar messages [ 7430.988568] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7433.439618] Lustre: Failing over lustre-MDT0000 [ 7433.703509] Lustre: server umount lustre-MDT0000 complete [ 7439.841728] LustreError: 165958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7439.868784] LustreError: 165958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 101 previous similar messages [ 7450.576052] Lustre: 3457:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787737313/real 1787737313] req@ffff94b24d372a00 x1874570750912896/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787737329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7450.636326] Lustre: 3457:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [ 7460.840506] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b36bccca80 x1874570750920448/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7461.538341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7461.546282] Lustre: Skipped 5 previous similar messages [ 7461.607197] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7461.610868] Lustre: Skipped 5 previous similar messages [ 7461.921144] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7461.931293] Lustre: Skipped 5 previous similar messages [ 7461.975689] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7461.978625] LustreError: 164997:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b24855ed80 x1874570739593472/t339302416941(339302416941) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:31/0 lens 520/664 e 0 to 0 dl 1787737351 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7466.481421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7466.486791] Lustre: Skipped 19 previous similar messages [ 7466.997463] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7477.375358] Lustre: lustre-MDT0000: Client 651c117a-a2a9-4aa4-8c7f-122b99d35934 (at 192.168.201.49@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7477.398956] Lustre: 164996:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b252e98a80 x1874570739593472/t339302416941(339302416941) o101->651c117a-a2a9-4aa4-8c7f-122b99d35934@192.168.201.49@tcp:47/0 lens 520/3488 e 0 to 0 dl 1787737367 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7477.489558] Lustre: lustre-MDT0000: Recovery over after 0:16, of 2 clients 2 recovered and 0 were evicted. [ 7477.503169] Lustre: Skipped 5 previous similar messages [ 7477.546043] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6155 to 0x2c0000401:6241) [ 7477.550655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6273) [ 7482.799976] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7484.132257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7493.274653] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 05:42:50 (1787737370) [ 7497.198535] Lustre: Failing over lustre-OST0000 [ 7497.220249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7497.245245] Lustre: Skipped 22 previous similar messages [ 7497.249480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7497.262463] Lustre: Skipped 2 previous similar messages [ 7497.476564] Lustre: server umount lustre-OST0000 complete [ 7501.429828] Lustre: Failing over lustre-MDT0000 [ 7501.587327] Lustre: server umount lustre-MDT0000 complete [ 7518.687697] Lustre: 3454:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787737381/real 1787737381] req@ffff94b350332a00 x1874570750950144/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787737397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7518.724830] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7518.736477] LustreError: Skipped 4 previous similar messages [ 7528.928337] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b24d32aa00 x1874570750956928/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7528.947859] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 7534.051078] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7534.781272] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6155 to 0x2c0000401:6273) [ 7543.099357] Lustre: lustre-OST0000: Denying connection for new client 2b72bbbc-bb9e-4028-9487-638cf47a93e0 (at 192.168.201.49@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 7546.898622] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7546.994668] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6305) [ 7557.164783] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 05:43:54 (1787737434) [ 7558.321721] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7558.326094] LustreError: 8104:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b3485f8700 x1874570750986368/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:128/0 lens 1312/4320 e 0 to 0 dl 1787737448 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7561.770394] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7563.733862] Lustre: Failing over lustre-MDT0000 [ 7564.040555] Lustre: server umount lustre-MDT0000 complete [ 7565.801847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7565.812787] LustreError: Skipped 4 previous similar messages [ 7587.204589] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7588.016100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6337) [ 7588.022806] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6305) [ 7597.181141] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7598.973897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7608.989169] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 05:44:46 (1787737486) [ 7610.576968] LustreError: 156274:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b3485fb100 x1874570751026688/t360777252878(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:180/0 lens 2520/4320 e 0 to 0 dl 1787737500 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7614.272666] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7618.523161] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7620.585609] Lustre: Failing over lustre-MDT0001 [ 7620.598497] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 7625.299806] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 7625.308623] Lustre: Skipped 3 previous similar messages [ 7625.944968] Lustre: server umount lustre-MDT0001 complete [ 7649.970562] Lustre: 164998:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b377f3df80 x1874570740815872/t17179869902(0) o36->2b72bbbc-bb9e-4028-9487-638cf47a93e0@192.168.201.49@tcp:244/0 lens 560/2880 e 0 to 0 dl 1787737564 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7649.987162] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1228 to 0x280000400:1249) [ 7650.000368] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1196 to 0x2c0000400:1217) [ 7650.022175] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7660.756073] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7662.412339] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7671.471685] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 05:45:48 (1787737548) [ 7672.882037] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 7672.887917] Lustre: Skipped 1 previous similar message [ 7676.152617] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7679.938398] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7681.771545] Lustre: Failing over lustre-MDT0000 [ 7681.989103] Lustre: server umount lustre-MDT0000 complete [ 7688.163016] Lustre: 174370:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787737551/real 1787737551] req@ffff94b375992a00 x1874570751064704/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1787737567 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7688.191635] Lustre: 174370:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7706.008396] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7706.715316] Lustre: 176211:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 7706.724268] Lustre: 176211:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 51 previous similar messages [ 7707.123206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6369) [ 7707.125949] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6337) [ 7715.594764] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7717.114870] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7720.814325] Lustre: Failing over lustre-MDT0001 [ 7721.031783] Lustre: server umount lustre-MDT0001 complete [ 7743.028109] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7744.186878] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1228 to 0x2c0000400:1249) [ 7744.187621] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1260 to 0x280000400:1281) [ 7753.243831] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7755.437683] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7766.155542] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 05:47:23 (1787737643) [ 7767.774332] LustreError: 156274:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b37616d180 x1874570751136256/t365072220176(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:337/0 lens 2520/4320 e 0 to 0 dl 1787737657 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 7767.803932] LustreError: 156274:0:(ldlm_lib.c:3373:target_send_reply_msg()) Skipped 1 previous similar message [ 7774.415177] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7779.487390] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7781.942838] Lustre: Failing over lustre-MDT0000 [ 7782.317685] Lustre: server umount lustre-MDT0000 complete [ 7786.917538] LustreError: 6434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787737665 with bad export cookie 15780213208399789370 [ 7786.926526] Lustre: Failing over lustre-MDT0001 [ 7786.934789] LustreError: 6434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7786.951699] LustreError: 178946:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0xa:0x0].0xf7117594 (ffff94b343bd9300) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7786.979333] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 7792.187543] LustreError: 178946:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff94b3485f8a80 x1874570751147904/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 7792.216368] LustreError: 178946:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 7792.767713] Lustre: server umount lustre-MDT0001 complete [ 7814.992781] LustreError: 179704:0:(llog.c:1655:llog_backup()) MGC192.168.201.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7815.002647] Lustre: 179704:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7831.009654] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b37f2f9180 x1874570751150464/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7837.149220] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7838.013631] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7839.580717] Lustre: 179713:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b37e14f800 x1874570740891648/t25769803846(0) o36->2b72bbbc-bb9e-4028-9487-638cf47a93e0@192.168.201.49@tcp:434/0 lens 560/2880 e 0 to 0 dl 1787737754 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7839.581914] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1260 to 0x2c0000400:1281) [ 7839.624262] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1292 to 0x280000400:1313) [ 7839.795119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6401) [ 7839.810751] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6369) [ 7849.461528] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7851.429463] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7853.657545] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7864.444623] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 05:49:02 (1787737742) [ 7865.736157] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7865.738210] Lustre: Skipped 1 previous similar message [ 7872.319550] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7874.333297] Lustre: Failing over lustre-MDT0000 [ 7874.592664] Lustre: server umount lustre-MDT0000 complete [ 7881.805544] Lustre: lustre-MDT0001: Client 2b72bbbc-bb9e-4028-9487-638cf47a93e0 (at 192.168.201.49@tcp) reconnecting [ 7881.809604] Lustre: Skipped 2 previous similar messages [ 7881.843902] Lustre: 179744:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b37ed1ed80 x1874570740929280/t30064771142(0) o36->2b72bbbc-bb9e-4028-9487-638cf47a93e0@192.168.201.49@tcp:451/0 lens 560/2880 e 0 to 0 dl 1787737771 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7898.908253] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7899.337474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6401) [ 7899.338857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6433) [ 7912.378917] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7915.035666] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7925.462392] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 05:50:02 (1787737802) [ 7926.898653] LustreError: 179744:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b37e83c700 x1874570740963456/t30064771213(0) o36->2b72bbbc-bb9e-4028-9487-638cf47a93e0@192.168.201.49@tcp:496/0 lens 560/448 e 0 to 0 dl 1787737816 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7926.934593] LustreError: 179744:0:(ldlm_lib.c:3373:target_send_reply_msg()) Skipped 1 previous similar message [ 7931.119321] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7933.375573] Lustre: Failing over lustre-MDT0001 [ 7933.636859] Lustre: server umount lustre-MDT0001 complete [ 7952.974028] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (not set up) [ 7952.982284] Lustre: Skipped 5 previous similar messages [ 7958.109949] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7958.679089] Lustre: 180773:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b3479e1500 x1874570740963456/t30064771213(0) o36->2b72bbbc-bb9e-4028-9487-638cf47a93e0@192.168.201.49@tcp:528/0 lens 560/2880 e 0 to 0 dl 1787737848 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 7958.754723] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1334 to 0x280000400:1377) [ 7958.759927] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1302 to 0x2c0000400:1345) [ 7969.103532] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7970.834933] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7981.639807] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 05:50:58 (1787737858) [ 7990.162693] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7994.382491] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7996.735149] Lustre: Failing over lustre-MDT0000 [ 7997.073803] Lustre: server umount lustre-MDT0000 complete [ 7998.028726] Lustre: lustre-MDT0001: Client 2b72bbbc-bb9e-4028-9487-638cf47a93e0 (at 192.168.201.49@tcp) reconnecting [ 7998.063537] Lustre: 179744:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b37eb4aa00 x1874570740996096/t34359738438(0) o36->2b72bbbc-bb9e-4028-9487-638cf47a93e0@192.168.201.49@tcp:568/0 lens 560/2880 e 0 to 0 dl 1787737888 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8015.337224] Lustre: 3457:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787737878/real 1787737878] req@ffff94b37e83c700 x1874570751285120/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787737894 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8015.370129] Lustre: 3457:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8026.592675] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b37e83c000 x1874570751294336/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8032.103840] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8032.434273] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6433) [ 8032.436456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6465) [ 8042.189753] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8044.029862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8048.472508] Lustre: Failing over lustre-MDT0001 [ 8048.708555] Lustre: server umount lustre-MDT0001 complete [ 8049.233914] LustreError: 180773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8049.255457] LustreError: 180773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 225 previous similar messages [ 8068.122248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8068.128624] Lustre: Skipped 11 previous similar messages [ 8068.168079] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8068.178726] Lustre: Skipped 11 previous similar messages [ 8069.308438] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8069.317142] Lustre: Skipped 11 previous similar messages [ 8072.807564] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8073.198562] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 8073.205484] Lustre: Skipped 39 previous similar messages [ 8073.351628] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1356 to 0x2c0000400:1377) [ 8073.359714] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1388 to 0x280000400:1409) [ 8081.822180] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8083.643249] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8094.205741] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 05:52:51 (1787737971) [ 8101.546444] LustreError: 187496:0:(osd_handler.c:715:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 8101.553341] LustreError: 187496:0:(osd_handler.c:715:osd_ro()) Skipped 11 previous similar messages [ 8102.652070] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8107.364850] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8109.459528] Lustre: Failing over lustre-MDT0000 [ 8109.724820] Lustre: server umount lustre-MDT0000 complete [ 8111.763523] Lustre: 179712:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b3797f2300 x1874570741041152/t38654705734(0) o36->2b72bbbc-bb9e-4028-9487-638cf47a93e0@192.168.201.49@tcp:681/0 lens 560/2880 e 0 to 0 dl 1787738001 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8114.144947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.168810] Lustre: Skipped 40 previous similar messages [ 8114.461896] LustreError: 154740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787737993 with bad export cookie 15780213208399803083 [ 8114.466190] Lustre: Failing over lustre-MDT0001 [ 8114.477031] LustreError: 154740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8114.858628] Lustre: server umount lustre-MDT0001 complete [ 8135.359406] LustreError: 188775:0:(llog.c:1655:llog_backup()) MGC192.168.201.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8135.371424] Lustre: 188775:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8146.810632] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8147.278248] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8147.605790] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8147.619852] Lustre: Skipped 12 previous similar messages [ 8147.670830] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1388 to 0x2c0000400:1409) [ 8147.671185] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1420 to 0x280000400:1441) [ 8147.980597] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6465) [ 8147.981147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6497) [ 8158.918718] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8160.755297] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8162.499805] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8172.772397] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 05:54:09 (1787738049) [ 8174.723890] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8174.726473] Lustre: Skipped 3 previous similar messages [ 8180.128638] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8182.852326] Lustre: Failing over lustre-MDT0001 [ 8182.865972] LustreError: 190423:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x240001f59:0xa9:0x0].0x0 (ffff94b375639000) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8182.911666] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 8183.786859] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8183.794214] LustreError: Skipped 8 previous similar messages [ 8188.300349] Lustre: server umount lustre-MDT0001 complete [ 8213.537431] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8213.690129] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1452 to 0x280000400:1473) [ 8213.694473] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1420 to 0x2c0000400:1441) [ 8213.713327] Lustre: 188788:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b2512a7480 x1874570741081472/t42949673032(0) o36->2b72bbbc-bb9e-4028-9487-638cf47a93e0@192.168.201.49@tcp:53/0 lens 496/2888 e 0 to 0 dl 1787738128 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8223.908586] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8225.482409] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8234.115691] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 05:55:11 (1787738111) [ 8235.684601] LustreError: 154689:0:(ldlm_lib.c:3373:target_send_reply_msg()) @@@ dropping reply req@ffff94b3759edc00 x1874570751445248/t382252089376(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:50/0 lens 1744/4320 e 0 to 0 dl 1787738125 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8235.713155] LustreError: 154689:0:(ldlm_lib.c:3373:target_send_reply_msg()) Skipped 3 previous similar messages [ 8240.180744] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8242.463108] Lustre: Failing over lustre-MDT0000 [ 8242.770214] Lustre: server umount lustre-MDT0000 complete [ 8260.575450] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8260.587720] LustreError: Skipped 6 previous similar messages [ 8270.815455] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b375bf2680 x1874570751457280/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8276.108459] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8276.599707] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6497) [ 8276.604240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6529) [ 8285.535539] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8287.234293] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8296.275332] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 05:56:13 (1787738173) [ 8302.267970] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8306.062412] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8307.872677] Lustre: Failing over lustre-MDT0000 [ 8308.216860] Lustre: server umount lustre-MDT0000 complete [ 8331.340438] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8332.382063] Lustre: 194357:0:(ldlm_lib.c:2110:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 8332.401124] Lustre: 194357:0:(ldlm_lib.c:2110:extend_recovery_timer()) Skipped 16 previous similar messages [ 8332.550629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6561) [ 8332.564195] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6529) [ 8340.869190] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8342.821834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8346.936956] Lustre: Failing over lustre-MDT0001 [ 8347.137521] Lustre: server umount lustre-MDT0001 complete [ 8371.426984] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8372.358388] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1420 to 0x2c0000400:1473) [ 8372.359265] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1452 to 0x280000400:1505) [ 8381.961392] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8384.036281] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8393.682403] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 05:57:51 (1787738271) [ 8399.398479] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8404.818730] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8406.606598] Lustre: Failing over lustre-MDT0000 [ 8406.911450] Lustre: server umount lustre-MDT0000 complete [ 8410.540310] Lustre: Failing over lustre-MDT0001 [ 8410.542043] LustreError: 6433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787738289 with bad export cookie 15780213208399813422 [ 8410.547756] LustreError: 197074:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0x1f:0x0].0xf7117594 (ffff94b3479d3500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8410.557312] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.49@tcp (stopping) [ 8410.557322] Lustre: Skipped 5 previous similar messages [ 8410.557378] LustreError: 6433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8410.604631] LustreError: 197074:0:(ldlm_resource.c:1207:ldlm_resource_complain()) Skipped 2 previous similar messages [ 8416.078775] Lustre: server umount lustre-MDT0001 complete [ 8435.628546] LustreError: 197821:0:(llog.c:1655:llog_backup()) MGC192.168.201.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8435.636603] Lustre: 197821:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8435.681642] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xdafe944e4e9924ce [ 8441.072170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6561) [ 8441.079272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6593) [ 8441.129419] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8441.267768] Lustre: 197834:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b248815f80 x1874570741133184/t51539607554(0) o36->2b72bbbc-bb9e-4028-9487-638cf47a93e0@192.168.201.49@tcp:281/0 lens 496/2888 e 0 to 0 dl 1787738356 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8441.274544] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1452 to 0x280000400:1537) [ 8441.277722] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1420 to 0x2c0000400:1505) [ 8441.695084] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8451.356682] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8452.873695] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8454.473936] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8463.392921] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 05:59:00 (1787738340) [ 8471.168877] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8473.266095] Lustre: Failing over lustre-MDT0000 [ 8473.580410] Lustre: server umount lustre-MDT0000 complete [ 8481.868080] Lustre: lustre-MDT0001: Client 2b72bbbc-bb9e-4028-9487-638cf47a93e0 (at 192.168.201.49@tcp) reconnecting [ 8481.883596] Lustre: Skipped 1 previous similar message [ 8492.497320] Lustre: 3455:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787738355/real 1787738355] req@ffff94b367073480 x1874570751600128/t0(0) o400->MGC192.168.201.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787738371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8492.521296] Lustre: 3455:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 8502.753129] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b37717ce00 x1874570751609600/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8508.579126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6625) [ 8508.587497] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6593) [ 8508.797421] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8518.938925] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8520.900874] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8531.690218] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 06:00:09 (1787738409) [ 8538.040743] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8540.480409] Lustre: Failing over lustre-MDT0001 [ 8541.087133] Lustre: server umount lustre-MDT0001 complete [ 8563.362065] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8564.918768] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1420 to 0x2c0000400:1537) [ 8564.919109] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1452 to 0x280000400:1569) [ 8572.002344] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8573.429450] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8582.447197] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 06:01:00 (1787738460) [ 8588.162268] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8592.621250] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8594.648576] Lustre: Failing over lustre-MDT0000 [ 8595.253223] Lustre: server umount lustre-MDT0000 complete [ 8627.444650] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6657) [ 8627.445880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6625) [ 8627.790906] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8636.816739] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8638.500879] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8642.517917] Lustre: Failing over lustre-MDT0001 [ 8642.535911] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8642.545523] Lustre: Skipped 6 previous similar messages [ 8642.821975] Lustre: server umount lustre-MDT0001 complete [ 8652.786123] LustreError: 198929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8652.811436] LustreError: 198929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 215 previous similar messages [ 8664.981483] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8666.214395] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1420 to 0x2c0000400:1569) [ 8666.214463] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1452 to 0x280000400:1601) [ 8675.008450] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8676.609638] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8685.239760] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 06:02:42 (1787738562) [ 8690.185897] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8693.761963] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8695.552051] Lustre: Failing over lustre-MDT0000 [ 8695.720218] Lustre: server umount lustre-MDT0000 complete [ 8698.993860] LustreError: 129665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787738577 with bad export cookie 15780213208399820905 [ 8698.998426] Lustre: Failing over lustre-MDT0001 [ 8699.004327] LustreError: 129665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8699.277679] Lustre: server umount lustre-MDT0001 complete [ 8717.280253] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.291189] Lustre: Skipped 42 previous similar messages [ 8717.479609] LustreError: 206922:0:(llog.c:1655:llog_backup()) MGC192.168.201.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8717.484387] Lustre: 206922:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8723.425117] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b37eb30e00 x1874570751747840/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8723.774528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8723.777834] Lustre: Skipped 12 previous similar messages [ 8723.806423] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8723.814626] Lustre: Skipped 12 previous similar messages [ 8727.203563] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8727.213847] Lustre: Skipped 13 previous similar messages [ 8727.217059] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 8727.229448] Lustre: Skipped 43 previous similar messages [ 8727.832794] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8728.149408] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1452 to 0x280000400:1633) [ 8728.150452] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1420 to 0x2c0000400:1601) [ 8728.178578] Lustre: 206924:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff94b37f3d8000 x1874570741203712/t64424509442(0) o36->2b72bbbc-bb9e-4028-9487-638cf47a93e0@192.168.201.49@tcp:543/0 lens 496/2888 e 0 to 0 dl 1787738618 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8728.211033] Lustre: 206924:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 3 previous similar messages [ 8728.443188] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8737.914598] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6275 to 0x2c0000401:6657) [ 8737.919476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6243 to 0x280000401:6689) [ 8743.481191] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8744.988377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8746.496919] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8754.475240] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 06:03:52 (1787738632) [ 8755.958927] Lustre: 208308:0:(genops.c:1774:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 2b72bbbc-bb9e-4028-9487-638cf47a93e0 at adminstrative request [ 8765.266224] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 06:04:03 (1787738643) [ 8772.459151] Lustre: Failing over lustre-MDT0000 [ 8772.820721] Lustre: server umount lustre-MDT0000 complete [ 8794.277782] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8795.912747] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8795.926382] Lustre: Skipped 13 previous similar messages [ 8795.973467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6740 to 0x280000401:6785) [ 8795.984515] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6709 to 0x2c0000401:6753) [ 8803.793473] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8805.375989] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8813.442136] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 06:04:51 (1787738691) [ 8827.033397] Lustre: Failing over lustre-OST0000 [ 8827.165956] Lustre: server umount lustre-OST0000 complete [ 8831.455843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8831.470970] LustreError: Skipped 8 previous similar messages [ 8853.101093] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8861.902639] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8863.820812] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8873.083054] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 06:05:50 (1787738750) [ 8878.001710] Lustre: Failing over lustre-MDT0000 [ 8878.276373] Lustre: server umount lustre-MDT0000 complete [ 8886.158961] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8886.165465] LustreError: Skipped 6 previous similar messages [ 8891.356826] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8892.105665] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6709 to 0x2c0000401:6785) [ 8892.114655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6740 to 0x280000401:6817) [ 8901.434967] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 06:06:18 (1787738778) [ 8905.487299] LustreError: 212750:0:(osd_handler.c:715:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 8905.494463] LustreError: 212750:0:(osd_handler.c:715:osd_ro()) Skipped 13 previous similar messages [ 8906.624476] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8909.847543] Lustre: Failing over lustre-OST0000 [ 8909.902468] Lustre: server umount lustre-OST0000 complete [ 8932.938557] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8942.198758] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8943.848748] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8952.113518] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 06:07:09 (1787738829) [ 8956.581856] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8960.165443] Lustre: Failing over lustre-OST0000 [ 8960.230205] Lustre: server umount lustre-OST0000 complete [ 8982.211863] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.49@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:6819 extent [0-1048575]: client csum 7e830193, server csum 70e0157 [ 8986.309207] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8995.039656] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8996.645866] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9004.601792] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 06:08:02 (1787738882) [ 9007.800951] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9010.790795] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9015.971334] Lustre: Failing over lustre-MDT0000 [ 9016.273419] Lustre: server umount lustre-MDT0000 complete [ 9019.632232] LustreError: 6433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787738898 with bad export cookie 15780213208399848163 [ 9019.641139] Lustre: Failing over lustre-OST0000 [ 9019.642195] LustreError: 6433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9019.695891] Lustre: server umount lustre-OST0000 complete [ 9044.959806] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b251cd3100 x1874570751957632/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9048.396043] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9051.167553] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6824 to 0x2c0000401:6849) [ 9070.710282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6820 to 0x280000401:6849) [ 9070.882918] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9084.996190] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 06:09:22 (1787738962) [ 9100.823358] Lustre: Failing over lustre-OST0000 [ 9100.923801] Lustre: server umount lustre-OST0000 complete [ 9104.969974] Lustre: Failing over lustre-MDT0000 [ 9105.205032] Lustre: server umount lustre-MDT0000 complete [ 9125.829622] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9127.581157] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6824 to 0x2c0000401:6881) [ 9138.186160] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9140.431721] Lustre: lustre-OST0000: Denying connection for new client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 9140.456450] Lustre: Skipped 1 previous similar message [ 9145.937828] Lustre: lustre-OST0000: Denying connection for new client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 9151.065164] Lustre: lustre-OST0000: Denying connection for new client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 9161.301879] Lustre: lustre-OST0000: Denying connection for new client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:43 [ 9161.334391] Lustre: Skipped 1 previous similar message [ 9181.785478] Lustre: lustre-OST0000: Denying connection for new client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:22 [ 9181.807895] Lustre: Skipped 3 previous similar messages [ 9204.506436] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9204.508332] Lustre: 220646:0:(genops.c:1601:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 1cb3e0d6-cf2e-4ea4-bf90-944ed496c56c@ [ 9204.548506] Lustre: 220646:0:(genops.c:1601:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 9204.574936] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9204.689968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6860 to 0x280000401:6881) [ 9211.249587] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 60 sec [ 9226.790852] Lustre: DEBUG MARKER: free_before: 7520256 free_after: 7519232 [ 9234.397642] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 06:11:51 (1787739111) [ 9239.186147] Lustre: Failing over lustre-OST0001 [ 9239.313631] Lustre: server umount lustre-OST0001 complete [ 9253.451640] LustreError: 38144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9253.469272] LustreError: 38144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 164 previous similar messages [ 9265.558195] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9274.651820] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 06:12:32 (1787739152) [ 9277.508050] Lustre: Failing over lustre-OST0000 [ 9277.599589] Lustre: server umount lustre-OST0000 complete [ 9296.121867] LustreError: 223724:0:(ldlm_lib.c:2930:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9296.129869] LustreError: 223724:0:(ldlm_lib.c:2930:target_recovery_thread()) Skipped 80 previous similar messages [ 9300.616147] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9302.495688] Lustre: *** cfs_fail_loc=715, val=40*** [ 9312.385205] Lustre: lustre-OST0000: Client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp) reconnected, waiting for 3 clients in recovery for 1:24 [ 9312.735140] Lustre: 3453:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787739175/real 1787739175] req@ffff94b377648a80 x1874570752096512/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787739191 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 9312.765538] Lustre: 3453:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 9318.880953] Lustre: *** cfs_fail_loc=715, val=40*** [ 9318.884603] Lustre: Skipped 2 previous similar messages [ 9328.107070] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 9328.128217] Lustre: Skipped 3 previous similar messages [ 9334.239175] Lustre: *** cfs_fail_loc=715, val=40*** [ 9334.241717] Lustre: Skipped 2 previous similar messages [ 9336.183220] LustreError: 223724:0:(ldlm_lib.c:2930:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9336.196541] LustreError: 223724:0:(ldlm_lib.c:2930:target_recovery_thread()) Skipped 80 previous similar messages [ 9336.276517] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9336.290549] Lustre: Skipped 31 previous similar messages [ 9342.341841] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9343.807599] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9353.426969] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 06:13:51 (1787739231) [ 9357.441539] Lustre: Failing over lustre-MDT0000 [ 9357.651170] Lustre: server umount lustre-MDT0000 complete [ 9358.309607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9358.336316] Lustre: Skipped 28 previous similar messages [ 9375.267463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9375.270533] Lustre: Skipped 12 previous similar messages [ 9375.366036] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9375.382442] Lustre: Skipped 12 previous similar messages [ 9376.806089] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9376.809296] Lustre: Skipped 11 previous similar messages [ 9379.341168] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9380.339929] LustreError: 225319:0:(ldlm_lib.c:2930:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 9382.879218] Lustre: *** cfs_fail_loc=715, val=80*** [ 9382.881128] Lustre: Skipped 2 previous similar messages [ 9392.212350] Lustre: lustre-MDT0000: Client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp) reconnected, waiting for 2 clients in recovery for 1:28 [ 9392.224960] Lustre: Skipped 1 previous similar message [ 9396.708482] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9398.752161] Lustre: *** cfs_fail_loc=715, val=80*** [ 9398.766382] Lustre: Skipped 1 previous similar message [ 9408.592844] Lustre: lustre-MDT0000: Client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp) reconnected, waiting for 2 clients in recovery for 1:11 [ 9412.065925] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9414.623298] Lustre: *** cfs_fail_loc=715, val=80*** [ 9414.631407] Lustre: Skipped 1 previous similar message [ 9424.970228] Lustre: lustre-MDT0000: Client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 9428.452217] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9431.008220] Lustre: *** cfs_fail_loc=715, val=80*** [ 9431.014821] Lustre: Skipped 1 previous similar message [ 9440.333291] Lustre: lustre-MDT0000: Client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp) reconnected, waiting for 2 clients in recovery for 0:40 [ 9444.831707] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9456.716728] Lustre: lustre-MDT0000: Client 11127da8-8dc1-485d-afe6-b9df6c980805 (at 192.168.201.49@tcp) reconnected, waiting for 2 clients in recovery for 0:23 [ 9460.194747] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9460.351258] LustreError: 225319:0:(ldlm_lib.c:2930:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9460.419755] Lustre: 225319:0:(ldlm_lib.c:2976:target_recovery_thread()) too long recovery - read logs [ 9460.443569] LustreError: dumping log to /tmp/lustre-log.1787739339.225319 [ 9460.569211] Lustre: lustre-MDT0000: Recovery over after 1:24, of 2 clients 2 recovered and 0 were evicted. [ 9460.572453] Lustre: Skipped 10 previous similar messages [ 9460.600455] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6894 to 0x2c0000401:6913) [ 9460.607429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6895 to 0x280000401:6913) [ 9466.902509] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9468.526980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9479.331784] Lustre: DEBUG MARKER: == replay-single test complete, duration 9194 sec ======== 06:15:56 (1787739356) [ 9481.331152] Lustre: DEBUG MARKER: === replay-single: start cleanup 06:15:58 (1787739358) === [ 9495.661963] Lustre: DEBUG MARKER: === replay-single: finish cleanup 06:16:12 (1787739372) === [ 9498.448217] Lustre: Failing over lustre-MDT0000 [ 9498.852521] Lustre: server umount lustre-MDT0000 complete [ 9517.549095] LustreError: MGC192.168.201.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9517.558771] LustreError: Skipped 3 previous similar messages [ 9527.777556] LustreError: 3453:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff94b3759ed880 x1874570752276224/t0(0) o250->MGC192.168.201.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9532.804416] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9533.606329] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6894 to 0x2c0000401:6945) [ 9533.611269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6895 to 0x280000401:6945) [ 9544.704789] Lustre: DEBUG MARKER: oleg149-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9546.606538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9553.892284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9553.897187] Lustre: Skipped 7 previous similar messages [ 9556.171471] Lustre: server umount lustre-MDT0000 complete [ 9560.696421] LustreError: 129665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787739439 with bad export cookie 15780213208399882092 [ 9560.721196] LustreError: 129665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9561.045313] Lustre: server umount lustre-MDT0001 complete [ 9574.267269] Lustre: server umount lustre-OST0000 complete [ 9588.931208] Lustre: server umount lustre-OST0001 complete [ 9602.986947] Lustre: DEBUG MARKER: oleg149-server.virtnet: executing unload_modules_local [ 9605.794408] Key type lgssc unregistered [ 9606.152810] LNet: 228858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9606.166085] LNetError: 228858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9607.207996] LNet: Removed LNI 192.168.201.149@tcp [ 9607.966143] Key type .llcrypt unregistered [ 9607.967752] Key type ._llcrypt unregistered