[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508766360 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003093] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.006006] kvm-guest: setup PV IPIs [ 0.008617] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011130] LSM: Security Framework initializing [ 0.012041] Yama: becoming mindful. [ 0.013029] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.022781] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027242] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028120] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030072] *** VALIDATE tmpfs *** [ 0.031357] *** VALIDATE proc *** [ 0.032203] *** VALIDATE cgroup *** [ 0.033005] *** VALIDATE cgroup2 *** [ 0.034227] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037021] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041000] debug: unmapping init [mem 0xffffffffa2459000-0xffffffffa2460fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043656] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044015] ... version: 2 [ 0.045007] ... bit width: 48 [ 0.046006] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048009] ... max period: 00007fffffffffff [ 0.049008] ... fixed-purpose events: 3 [ 0.050008] ... event mask: 000000070000000f [ 0.051231] rcu: Hierarchical SRCU implementation. [ 0.053355] smp: Bringing up secondary CPUs ... [ 0.054434] x86: Booting SMP configuration: [ 0.055013] .... node #0, CPUs: #1 #2 #3 [ 0.067073] smp: Brought up 1 node, 4 CPUs [ 0.069008] smpboot: Max logical packages: 1 [ 0.070016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151273] node 0 deferred pages initialised in 79ms [ 0.155352] devtmpfs: initialized [ 0.156238] x86/mm: Memory block size: 128MB [ 0.159082] gcov: version magic: 0x41383552 [ 0.161390] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164324] pinctrl core: initialized pinctrl subsystem [ 0.166196] [ 0.166792] ************************************************************* [ 0.170013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172011] ** ** [ 0.174008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175010] ** ** [ 0.177010] ** This means that this kernel is built to expose internal ** [ 0.179010] ** IOMMU data structures, which may compromise security on ** [ 0.182013] ** your system. ** [ 0.184009] ** ** [ 0.186010] ** If you see this message and you are not debugging the ** [ 0.188009] ** kernel, report this immediately to your vendor! ** [ 0.190010] ** ** [ 0.193010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195011] ************************************************************* [ 0.197770] NET: Registered protocol family 16 [ 0.199448] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.205061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.210483] cpuidle: using governor menu [ 0.212984] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215384] PCI: Using configuration type 1 for base access [ 0.217131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225368] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228199] cryptd: max_cpu_qlen set to 1000 [ 0.230622] ACPI: Added _OSI(Module Device) [ 0.232020] ACPI: Added _OSI(Processor Device) [ 0.233020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241537] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.247242] ACPI: Interpreter enabled [ 0.248049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.250018] ACPI: Using IOAPIC for interrupt routing [ 0.251130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255650] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273050] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277125] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288234] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.297431] acpiphp: Slot [2] registered [ 0.299153] acpiphp: Slot [5] registered [ 0.300086] acpiphp: Slot [6] registered [ 0.302106] acpiphp: Slot [7] registered [ 0.303077] acpiphp: Slot [8] registered [ 0.304116] acpiphp: Slot [9] registered [ 0.306119] acpiphp: Slot [10] registered [ 0.308164] acpiphp: Slot [3] registered [ 0.310091] acpiphp: Slot [4] registered [ 0.312122] acpiphp: Slot [11] registered [ 0.313099] acpiphp: Slot [12] registered [ 0.315092] acpiphp: Slot [13] registered [ 0.316070] acpiphp: Slot [14] registered [ 0.318074] acpiphp: Slot [15] registered [ 0.319117] acpiphp: Slot [16] registered [ 0.320073] acpiphp: Slot [17] registered [ 0.322065] acpiphp: Slot [18] registered [ 0.323114] acpiphp: Slot [19] registered [ 0.325086] acpiphp: Slot [20] registered [ 0.327085] acpiphp: Slot [21] registered [ 0.328116] acpiphp: Slot [22] registered [ 0.329090] acpiphp: Slot [23] registered [ 0.331114] acpiphp: Slot [24] registered [ 0.332083] acpiphp: Slot [25] registered [ 0.333123] acpiphp: Slot [26] registered [ 0.335075] acpiphp: Slot [27] registered [ 0.336075] acpiphp: Slot [28] registered [ 0.338165] acpiphp: Slot [29] registered [ 0.339070] acpiphp: Slot [30] registered [ 0.340078] acpiphp: Slot [31] registered [ 0.342055] PCI host bridge to bus 0000:00 [ 0.345031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.347022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.349065] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.352021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.354019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.356025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.360944] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.364217] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.373013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.378475] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.381019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.383015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.385016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.388570] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.390765] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.395040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.399935] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.408013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.424012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.430013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.437310] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.452015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.467017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.508016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.528621] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.551014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.568014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.603013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.613089] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.618012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.622015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.634012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.640738] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.645953] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.649014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.667013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.676145] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.680941] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.684016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.695016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.706345] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.714016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.721012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.735014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.746117] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.748542] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.751523] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.754415] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.761420] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.774285] iommu: Default domain type: Passthrough [ 0.776503] SCSI subsystem initialized [ 0.777160] ACPI: bus type USB registered [ 0.779088] usbcore: registered new interface driver usbfs [ 0.780084] usbcore: registered new interface driver hub [ 0.782068] usbcore: registered new device driver usb [ 0.784163] pps_core: LinuxPPS API ver. 1 registered [ 0.786011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.789049] PTP clock support registered [ 0.792093] EDAC MC: Ver: 3.0.0 [ 0.794012] PCI: Using ACPI for IRQ routing [ 0.796917] NetLabel: Initializing [ 0.798011] NetLabel: domain hash size = 128 [ 0.800014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.803073] NetLabel: unlabeled traffic allowed by default [ 0.806107] vgaarb: loaded [ 0.807233] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.808015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.809000] clocksource: Switched to clocksource kvm-clock [ 0.959339] VFS: Disk quotas dquot_6.6.0 [ 0.963461] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.967698] *** VALIDATE ramfs *** [ 0.969638] *** VALIDATE hugetlbfs *** [ 0.972265] pnp: PnP ACPI init [ 0.975574] pnp: PnP ACPI: found 6 devices [ 0.994840] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.000241] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.001976] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.003671] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.005606] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.007501] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.009836] NET: Registered protocol family 2 [ 1.011983] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.016426] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.019780] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.024593] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.027936] TCP: Hash tables configured (established 65536 bind 65536) [ 1.030537] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.032829] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.035379] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.037757] NET: Registered protocol family 1 [ 1.040435] RPC: Registered named UNIX socket transport module. [ 1.042201] RPC: Registered udp transport module. [ 1.043579] RPC: Registered tcp transport module. [ 1.044868] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.046873] NET: Registered protocol family 44 [ 1.048864] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.051461] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.053819] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.056725] PCI: CLS 0 bytes, default 64 [ 1.058528] Unpacking initramfs... [ 2.636818] debug: unmapping init [mem 0xffff8ed4bcc54000-0xffff8ed4bffbffff] [ 2.640784] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.642857] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.645540] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.454713] Initialise system trusted keyrings [ 3.456064] Key type blacklist registered [ 3.457834] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.466379] zbud: loaded [ 3.469165] *** VALIDATE nfs *** [ 3.470398] *** VALIDATE nfs4 *** [ 3.471774] pstore: using deflate compression [ 3.474842] Platform Keyring initialized [ 3.585361] NET: Registered protocol family 38 [ 3.586797] Key type asymmetric registered [ 3.588310] Asymmetric key parser 'x509' registered [ 3.589877] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.592546] io scheduler mq-deadline registered [ 3.593880] io scheduler kyber registered [ 3.595345] io scheduler bfq registered [ 3.607782] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.610655] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.614692] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.617780] ACPI: Power Button [PWRF] [ 3.625241] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.638560] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.662868] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.671961] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.701264] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.748378] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.787816] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.792606] Non-volatile memory driver v1.3 [ 3.794224] Linux agpgart interface v0.103 [ 3.822770] virtio_blk virtio1: [vda] 132376 512-byte logical blocks (67.8 MB/64.6 MiB) [ 3.824796] vda: detected capacity change from 0 to 67776512 [ 3.845709] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.848404] vdb: detected capacity change from 0 to 1073741824 [ 3.874428] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.877540] vdc: detected capacity change from 0 to 2621440000 [ 3.903279] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.905717] vdd: detected capacity change from 0 to 2621440000 [ 3.924482] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.927845] vde: detected capacity change from 0 to 4294967296 [ 3.943426] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.946481] vdf: detected capacity change from 0 to 4294967296 [ 3.957443] libphy: Fixed MDIO Bus: probed [ 3.961581] usbcore: registered new interface driver usbserial_generic [ 3.963348] usbserial: USB Serial support registered for generic [ 3.965409] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.969140] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.970491] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.972493] mousedev: PS/2 mouse device common for all mice [ 3.974887] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.977432] rtc_cmos 00:05: RTC can wake from S4 [ 3.981062] rtc_cmos 00:05: registered as rtc0 [ 3.982966] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.986343] intel_pstate: CPU model not supported [ 3.988499] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.993449] hid: raw HID events driver (C) Jiri Kosina [ 3.996128] usbcore: registered new interface driver usbhid [ 3.998290] usbhid: USB HID core driver [ 3.999785] drop_monitor: Initializing network drop monitor service [ 4.002316] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.003326] Initializing XFRM netlink socket [ 4.007741] NET: Registered protocol family 10 [ 4.010842] Segment Routing with IPv6 [ 4.012325] NET: Registered protocol family 17 [ 4.014312] mpls_gso: MPLS GSO support [ 4.018608] RAS: Correctable Errors collector initialized. [ 4.020718] AVX version of gcm_enc/dec engaged. [ 4.022314] AES CTR mode by8 optimization enabled [ 4.101676] sched_clock: Marking stable (4101516418, 0)->(5046708192, -945191774) [ 4.108567] registered taskstats version 1 [ 4.110718] Loading compiled-in X.509 certificates [ 4.113309] zswap: loaded using pool lzo/zbud [ 4.137807] Key type big_key registered [ 4.153316] Key type encrypted registered [ 4.156470] ima: No TPM chip found, activating TPM-bypass! [ 4.159502] ima: Allocated hash algorithm: sha1 [ 4.161992] ima: No architecture policies found [ 4.164610] evm: Initialising EVM extended attributes: [ 4.166551] evm: security.selinux [ 4.167757] evm: security.ima [ 4.169085] evm: security.capability [ 4.171219] evm: HMAC attrs: 0x1 [ 4.174446] rtc_cmos 00:05: setting system clock to 2026-04-02 21:50:27 UTC (1775166627) [ 4.181420] debug: unmapping init [mem 0xffffffffa3403000-0xffffffffa35fffff] [ 4.184720] debug: unmapping init [mem 0xffffffffa2182000-0xffffffffa2458fff] [ 4.192149] Write protecting the kernel read-only data: 28672k [ 4.198280] debug: unmapping init [mem 0xffffffffa0803000-0xffffffffa09fffff] [ 4.202016] debug: unmapping init [mem 0xffffffffa1114000-0xffffffffa11fffff] [ 4.243609] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.254120] systemd[1]: Detected virtualization kvm. [ 4.256728] systemd[1]: Detected architecture x86-64. [ 4.258974] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.284871] systemd[1]: No hostname configured. [ 4.290464] systemd[1]: Set hostname to . [ 4.292393] random: systemd: uninitialized urandom read (16 bytes read) [ 4.294623] systemd[1]: Initializing machine ID from random generator. [ 4.359524] random: ln: uninitialized urandom read (6 bytes read) [ 4.487272] random: systemd: uninitialized urandom read (16 bytes read) [ 4.490487] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.502886] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.514754] systemd[1]: Starting Create Volatile Files and Directories... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.584376] device-mapper: uevent: version 1.0.3 [ 5.586673] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.618609] virtio_net virtio0 ens2: renamed from eth0 [ 6.637649] random: fast init done [ 6.736921] scsi host0: ata_piix [ 6.766113] scsi host1: ata_piix [ 6.767494] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.769839] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.030996] random: crng init done [ 12.033639] random: 7 urandom warning(s) missed due to ratelimiting [ 14.836981] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.719879] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.447630] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.875688] SELinux: Disabled at runtime. [ 17.970083] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.979161] systemd[1]: Detected virtualization kvm. [ 17.981332] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.654323] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.656973] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.665290] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.669304] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.672388] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.681589] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.690792] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ OK [ 18.878622] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.479816] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.239345] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.373557] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.534225] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.884615] EDAC sbridge: Ver: 1.1.2 [ 24.078797] Key type dns_resolver registered [ 24.423813] NFS: Registering the id_resolver key type [ 24.425987] Key type id_resolver registered [ 24.427757] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg341-server login: [ 57.817150] libcfs: loading out-of-tree module taints kernel. [ 57.833222] Key type ._llcrypt registered [ 57.834973] Key type .llcrypt registered [ 57.879645] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_hostid [ 65.892595] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 66.516452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.525762] alg: No test for adler32 (adler32-zlib) [ 67.554344] Lustre: Lustre: Build Version: 2.17.51_23_g103a214 [ 67.944020] LNet: Added LNI 192.168.203.141@tcp [8/256/0/180] [ 69.576136] Key type lgssc registered [ 70.301919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 77.762618] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 95.957483] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing load_modules_local [ 101.797458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 101.811467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 102.930478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 102.956771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 103.005758] Lustre: lustre-MDT0000: new disk, initializing [ 103.051592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 103.067451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 104.750906] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.291458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 111.330779] Lustre: 6483:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 111.344857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 111.347766] Lustre: Skipped 1 previous similar message [ 111.394617] Lustre: lustre-MDT0001: new disk, initializing [ 111.423043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 111.435988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 111.441030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 113.057647] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 115.844346] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 119.770464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 119.869619] Lustre: lustre-OST0000: new disk, initializing [ 119.872523] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 119.895871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.979306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 121.985309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 122.017509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 122.316577] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.434045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 128.475935] Lustre: lustre-OST0001: new disk, initializing [ 128.477716] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 128.494020] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 130.461719] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 131.373421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 131.378580] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 131.391684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 136.400964] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 141.442157] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 148.819516] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing check_logdir /tmp/testlogs/ [ 150.684200] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing yml_node [ 152.541460] Lustre: DEBUG MARKER: Client: 2.17.51.23 [ 153.642586] Lustre: DEBUG MARKER: MDS: 2.17.51.23 [ 154.763427] Lustre: DEBUG MARKER: OSS: 2.17.51.23 [ 155.454484] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Thu Apr 2 17:52:58 EDT 2026 [ 162.864319] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 163.617264] Lustre: DEBUG MARKER: === replay-single: start setup 17:53:06 (1775166786) === [ 164.838121] hrtimer: interrupt took 7113936 ns [ 165.203419] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing check_config_client /mnt/lustre [ 176.774457] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 179.945683] Lustre: 13149:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 183.421868] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.981285] Lustre: DEBUG MARKER: === replay-single: finish setup 17:53:30 (1775166810) === [ 189.486660] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 17:53:31 (1775166811) [ 195.803323] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 197.515718] Lustre: Failing over lustre-MDT0000 [ 197.785746] Lustre: server umount lustre-MDT0000 complete [ 198.113040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 198.132252] Lustre: Skipped 1 previous similar message [ 203.234603] LustreError: 7504:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 203.242344] LustreError: 7504:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 9 previous similar messages [ 205.831504] LustreError: 6490:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 208.353092] LustreError: 6491:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 208.365614] LustreError: 6491:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 3 previous similar messages [ 210.955989] LustreError: 6492:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 214.497170] Lustre: 3642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775166821/real 1775166821] req@ffff8ed406e2b800 x1861397188267008/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775166837 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 214.511132] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 216.068123] LustreError: 6519:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 216.084272] LustreError: 6519:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 8 previous similar messages [ 216.384710] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 216.387418] LDISKFS-fs (dm-0): recovery complete [ 216.403470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.713084] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ed54196f100 x1861397188275456/t0(0) o250->MGC192.168.203.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 223.883109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.318637] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 226.470457] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.350713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 229.376637] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 231.798493] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 232.803915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 238.142655] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 17:54:20 (1775166860) [ 239.427845] Lustre: Failing over lustre-OST0000 [ 239.499406] Lustre: server umount lustre-OST0000 complete [ 239.586438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 239.593599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 239.603563] Lustre: Skipped 3 previous similar messages [ 239.606488] LustreError: 15175:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 239.625505] LustreError: 15175:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 11 previous similar messages [ 254.465266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 254.581714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.901898] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 255.933280] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 255.933359] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 255.947038] Lustre: Skipped 3 previous similar messages [ 257.876081] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.881429] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 263.876962] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 269.869500] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 17:54:52 (1775166892) [ 274.290533] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 275.479705] Lustre: Failing over lustre-MDT0000 [ 275.617491] Lustre: server umount lustre-MDT0000 complete [ 280.547046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.551722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 280.561479] LustreError: 6519:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.587698] LustreError: 6519:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 10 previous similar messages [ 292.984428] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 292.987195] LDISKFS-fs (dm-0): recovery complete [ 292.992944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 293.080544] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.232477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.554707] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 296.875755] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 296.883235] Lustre: lustre-MDT0000: Denying connection for new client c7e9a978-1a50-4e7a-9364-538635c55563 (at 192.168.203.41@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 298.467738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 298.472144] Lustre: Skipped 1 previous similar message [ 302.084757] Lustre: lustre-MDT0000: Denying connection for new client c7e9a978-1a50-4e7a-9364-538635c55563 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 307.206498] Lustre: lustre-MDT0000: Denying connection for new client c7e9a978-1a50-4e7a-9364-538635c55563 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 312.326159] Lustre: lustre-MDT0000: Denying connection for new client c7e9a978-1a50-4e7a-9364-538635c55563 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 317.445377] Lustre: lustre-MDT0000: Denying connection for new client c7e9a978-1a50-4e7a-9364-538635c55563 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 327.682867] Lustre: lustre-MDT0000: Denying connection for new client c7e9a978-1a50-4e7a-9364-538635c55563 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 327.694962] Lustre: Skipped 1 previous similar message [ 348.164475] Lustre: lustre-MDT0000: Denying connection for new client c7e9a978-1a50-4e7a-9364-538635c55563 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 348.188328] Lustre: Skipped 3 previous similar messages [ 366.500204] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 366.505188] Lustre: 17812:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d8c4a4a5-d068-4d54-9ce3-e63b61388490@ [ 366.526039] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 366.545681] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 366.546983] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 366.558770] Lustre: Skipped 2 previous similar messages [ 366.582599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 366.583290] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 374.615270] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 17:56:37 (1775166997) [ 379.285752] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 380.521947] Lustre: Failing over lustre-MDT0000 [ 380.682119] Lustre: server umount lustre-MDT0000 complete [ 381.922810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 381.929959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 381.941470] Lustre: Skipped 3 previous similar messages [ 381.945468] LustreError: 6496:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 381.959204] LustreError: 6496:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 8 previous similar messages [ 397.799659] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 397.802123] LDISKFS-fs (dm-0): recovery complete [ 397.808780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 397.876958] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 398.014773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.717661] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 402.140165] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 402.148997] Lustre: lustre-MDT0000: Denying connection for new client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 402.162742] Lustre: Skipped 3 previous similar messages [ 403.428402] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 469.000521] Lustre: lustre-MDT0000: Denying connection for new client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 469.010290] Lustre: Skipped 12 previous similar messages [ 472.500143] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 472.503655] Lustre: 19529:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c7e9a978-1a50-4e7a-9364-538635c55563@ [ 472.514371] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 472.524775] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 472.525266] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 472.539779] Lustre: Skipped 2 previous similar messages [ 472.558847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 472.559775] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 479.732233] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 17:58:22 (1775167102) [ 483.995710] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 485.005128] Lustre: Failing over lustre-MDT0000 [ 485.134576] Lustre: server umount lustre-MDT0000 complete [ 485.345950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 485.346528] LustreError: 6490:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.361202] Lustre: Skipped 6 previous similar messages [ 485.381918] LustreError: 6490:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 15 previous similar messages [ 501.951304] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 501.953761] LDISKFS-fs (dm-0): recovery complete [ 501.960760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 502.022289] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 502.158907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 504.244762] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 504.862347] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 507.363936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 507.399141] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 507.430677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 507.430951] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 509.159387] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 509.980884] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 514.502260] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 17:58:57 (1775167137) [ 517.521551] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 518.297063] Lustre: Failing over lustre-MDT0000 [ 518.500674] Lustre: server umount lustre-MDT0000 complete [ 522.723102] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.732281] Lustre: Skipped 3 previous similar messages [ 522.734036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 534.166450] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 534.168093] LDISKFS-fs (dm-0): recovery complete [ 534.176715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.258335] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.555224] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 536.426761] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 539.620832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 539.626486] Lustre: Skipped 3 previous similar messages [ 539.659422] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 539.689974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 539.690791] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 541.435691] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 542.201776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 546.412116] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 17:59:29 (1775167169) [ 549.830026] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 550.758982] Lustre: Failing over lustre-MDT0000 [ 550.878285] Lustre: server umount lustre-MDT0000 complete [ 554.976503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 554.980522] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.993519] Lustre: Skipped 2 previous similar messages [ 566.042061] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 566.044660] LDISKFS-fs (dm-0): recovery complete [ 566.055161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.114764] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 566.275499] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 567.876956] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 571.424972] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 571.456125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 571.458237] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:193) [ 573.109632] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 573.809727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 578.101501] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 18:00:00 (1775167200) [ 581.428332] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 582.321184] Lustre: Failing over lustre-MDT0000 [ 582.450764] Lustre: server umount lustre-MDT0000 complete [ 598.317393] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 598.319396] LDISKFS-fs (dm-0): recovery complete [ 598.325270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 598.400203] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.428593] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 602.115744] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 603.622786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 603.627179] Lustre: Skipped 9 previous similar messages [ 603.662670] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 603.682276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 603.682282] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 605.176954] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 605.930355] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 610.243677] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 18:00:33 (1775167233) [ 613.427907] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 614.405891] Lustre: Failing over lustre-MDT0000 [ 614.597659] Lustre: server umount lustre-MDT0000 complete [ 617.485511] LustreError: 9266:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.494841] LustreError: 9266:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 60 previous similar messages [ 618.976962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 618.977359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.991855] Lustre: Skipped 8 previous similar messages [ 630.424465] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 630.427117] LDISKFS-fs (dm-0): recovery complete [ 630.437544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.511300] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 630.623296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 630.628701] Lustre: Skipped 3 previous similar messages [ 630.646828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 632.375386] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 635.913638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 635.914425] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 637.334722] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 638.040832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 641.959348] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 18:01:04 (1775167264) [ 642.360549] Lustre: *** cfs_fail_loc=13b, val=315*** [ 642.362599] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 642.364691] LustreError: 6493:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed538c5c700 x1861397179917696/t38654705666(0) o35->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:706/0 lens 392/456 e 0 to 0 dl 1775167276 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 646.584660] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 647.346273] Lustre: Failing over lustre-MDT0000 [ 647.517990] Lustre: server umount lustre-MDT0000 complete [ 651.232986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 662.895204] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 662.897499] LDISKFS-fs (dm-0): recovery complete [ 662.904659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.114018] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 664.878856] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 668.157651] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 668.164342] Lustre: 6493:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed535e63b80 x1861397179917696/t38654705666(0) o35->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:732/0 lens 392/456 e 0 to 0 dl 1775167302 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 668.164600] Lustre: Skipped 1 previous similar message [ 668.199018] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 668.199064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 669.655303] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 670.347964] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 674.706844] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 18:01:37 (1775167297) [ 677.994390] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 678.808562] Lustre: Failing over lustre-MDT0000 [ 678.958436] Lustre: server umount lustre-MDT0000 complete [ 683.489594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 683.489606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 683.498705] Lustre: Skipped 6 previous similar messages [ 694.539835] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 694.541884] LDISKFS-fs (dm-0): recovery complete [ 694.547421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 694.629466] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 694.634377] LustreError: Skipped 1 previous similar message [ 694.746725] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 696.559911] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 699.397243] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 699.408306] Lustre: Skipped 2 previous similar messages [ 699.910846] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 699.910964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 701.523604] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 702.358933] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 706.670757] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 18:02:09 (1775167329) [ 709.854392] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 710.237282] Lustre: *** cfs_fail_loc=114, val=0*** [ 711.353368] Lustre: Failing over lustre-MDT0000 [ 711.551469] Lustre: server umount lustre-MDT0000 complete [ 726.459328] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 726.461265] LDISKFS-fs (dm-0): recovery complete [ 726.466091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 726.650853] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 728.256624] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 732.132053] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 732.135308] Lustre: Skipped 13 previous similar messages [ 732.164306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 732.164518] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 733.610597] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 734.259605] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 738.443589] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 18:02:41 (1775167361) [ 741.661279] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 742.117676] Lustre: *** cfs_fail_loc=128, val=0*** [ 743.235851] Lustre: Failing over lustre-MDT0000 [ 743.388490] Lustre: server umount lustre-MDT0000 complete [ 758.641637] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 758.644289] LDISKFS-fs (dm-0): recovery complete [ 758.653406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.825800] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 760.456321] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 763.960263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 763.962492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 765.456143] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 766.147422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 770.099260] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 18:03:12 (1775167392) [ 773.117661] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 773.934995] Lustre: Failing over lustre-MDT0000 [ 774.114142] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 774.117845] LustreError: Skipped 1 previous similar message [ 774.135146] Lustre: server umount lustre-MDT0000 complete [ 789.450907] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 789.452894] LDISKFS-fs (dm-0): recovery complete [ 789.463169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 789.636721] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 791.321161] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 795.239371] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 795.240446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 796.803503] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 797.481976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 801.683384] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 18:03:44 (1775167424) [ 804.990848] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 805.903421] Lustre: Failing over lustre-MDT0000 [ 806.044366] Lustre: server umount lustre-MDT0000 complete [ 821.403416] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 821.405627] LDISKFS-fs (dm-0): recovery complete [ 821.411778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 821.588318] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 823.244605] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 826.897316] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 826.901194] Lustre: Skipped 4 previous similar messages [ 826.917718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 826.917743] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 828.309367] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 828.985452] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 832.862727] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 18:04:15 (1775167455) [ 835.903399] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 838.881483] Lustre: Failing over lustre-MDT0000 [ 839.121616] Lustre: server umount lustre-MDT0000 complete [ 842.208922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.214232] Lustre: Skipped 19 previous similar messages [ 853.785905] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 853.790683] LDISKFS-fs (dm-0): recovery complete [ 853.800734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 853.854230] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 853.857894] LustreError: Skipped 4 previous similar messages [ 853.960058] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 855.428996] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 858.114548] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 858.118170] Lustre: Skipped 4 previous similar messages [ 859.807798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 859.807946] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 861.045089] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 861.645222] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 872.038875] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 18:04:54 (1775167494) [ 874.929128] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 875.679902] Lustre: Failing over lustre-MDT0000 [ 875.854018] Lustre: server umount lustre-MDT0000 complete [ 878.602496] LustreError: 9266:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 878.608901] LustreError: 9266:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 124 previous similar messages [ 890.306803] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 890.308027] LDISKFS-fs (dm-0): recovery complete [ 890.311395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.430970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.434591] Lustre: Skipped 7 previous similar messages [ 891.781554] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 895.495505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 895.495556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 896.895740] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 897.578848] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 903.405219] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 18:05:26 (1775167526) [ 906.406908] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 907.218850] Lustre: Failing over lustre-MDT0000 [ 907.427712] Lustre: server umount lustre-MDT0000 complete [ 910.816866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 910.820054] LustreError: Skipped 2 previous similar messages [ 922.025442] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 922.026826] LDISKFS-fs (dm-0): recovery complete [ 922.032636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 922.175565] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 922.178416] Lustre: Skipped 1 previous similar message [ 923.665507] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 927.241096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 927.241125] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 928.557348] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 929.163357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 932.883733] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 18:05:55 (1775167555) [ 935.741910] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 936.517194] Lustre: Failing over lustre-MDT0000 [ 936.690109] Lustre: server umount lustre-MDT0000 complete [ 951.367142] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 951.369018] LDISKFS-fs (dm-0): recovery complete [ 951.373447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.994474] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 956.925426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 956.925447] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 958.169749] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 958.762361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 962.433714] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 18:06:25 (1775167585) [ 965.306112] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 966.020325] Lustre: Failing over lustre-MDT0000 [ 966.224031] Lustre: server umount lustre-MDT0000 complete [ 980.510114] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 980.512324] LDISKFS-fs (dm-0): recovery complete [ 980.516574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.975503] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 986.131512] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 986.131560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 987.395557] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 987.971374] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 991.850915] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 18:06:54 (1775167614) [ 994.659462] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 995.349147] Lustre: Failing over lustre-MDT0000 [ 995.582288] Lustre: server umount lustre-MDT0000 complete [ 1010.354331] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1010.356197] LDISKFS-fs (dm-0): recovery complete [ 1010.360063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.020448] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1015.780303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1015.783325] Lustre: Skipped 35 previous similar messages [ 1015.819362] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1015.819367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1017.085236] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1017.680169] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1021.488056] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 18:07:24 (1775167644) [ 1024.449652] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1025.247990] Lustre: Failing over lustre-MDT0000 [ 1025.452258] Lustre: server umount lustre-MDT0000 complete [ 1040.332967] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1040.335278] LDISKFS-fs (dm-0): recovery complete [ 1040.341255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.160761] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1046.025359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1046.025378] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1047.458216] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1048.149121] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1051.938584] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 18:07:54 (1775167674) [ 1054.629413] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1055.357357] Lustre: Failing over lustre-MDT0000 [ 1055.554092] Lustre: server umount lustre-MDT0000 complete [ 1069.859558] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1069.861223] LDISKFS-fs (dm-0): recovery complete [ 1069.865773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.980420] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1069.983193] Lustre: Skipped 4 previous similar messages [ 1071.357177] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1075.217595] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1075.217614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1076.497925] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1077.129320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1080.853705] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 18:08:23 (1775167703) [ 1083.777385] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1084.545814] Lustre: Failing over lustre-MDT0000 [ 1084.763543] Lustre: server umount lustre-MDT0000 complete [ 1099.400895] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1099.402411] LDISKFS-fs (dm-0): recovery complete [ 1099.406105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.850220] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1104.874605] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1104.878391] Lustre: Skipped 8 previous similar messages [ 1104.894428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1104.894582] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1106.017628] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1106.565234] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1110.183696] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 18:08:53 (1775167733) [ 1113.041338] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1113.770150] Lustre: Failing over lustre-MDT0000 [ 1113.938211] Lustre: server umount lustre-MDT0000 complete [ 1115.104941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.110051] Lustre: Skipped 35 previous similar messages [ 1128.938156] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1128.939867] LDISKFS-fs (dm-0): recovery complete [ 1128.944453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.000302] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1129.004169] LustreError: Skipped 8 previous similar messages [ 1129.476328] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1129.481322] Lustre: Skipped 8 previous similar messages [ 1130.548607] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1134.585021] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1134.585040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1135.932770] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1136.617474] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1140.484488] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 18:09:23 (1775167763) [ 1143.589842] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1144.432101] Lustre: Failing over lustre-MDT0000 [ 1144.616533] Lustre: server umount lustre-MDT0000 complete [ 1159.087619] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1159.089327] LDISKFS-fs (dm-0): recovery complete [ 1159.092389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.676876] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1164.281617] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:897) [ 1164.281618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 1165.563365] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1166.148183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1169.783906] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 18:09:52 (1775167792) [ 1172.484782] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1173.196379] Lustre: Failing over lustre-MDT0000 [ 1173.361167] Lustre: server umount lustre-MDT0000 complete [ 1174.497432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1174.500152] LustreError: Skipped 8 previous similar messages [ 1187.791611] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1187.793989] LDISKFS-fs (dm-0): recovery complete [ 1187.800507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.366708] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1192.961091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1192.961092] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1194.072571] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1194.590521] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1198.024341] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 18:10:21 (1775167821) [ 1200.582234] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1201.287781] Lustre: Failing over lustre-MDT0000 [ 1201.459834] Lustre: server umount lustre-MDT0000 complete [ 1215.672515] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1215.673938] LDISKFS-fs (dm-0): recovery complete [ 1215.677221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.037870] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1221.123736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 1221.124428] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 1222.218263] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1222.732680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1226.157505] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 18:10:49 (1775167849) [ 1228.774555] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1229.423116] Lustre: Failing over lustre-MDT0000 [ 1229.601387] Lustre: server umount lustre-MDT0000 complete [ 1243.781062] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1243.783106] LDISKFS-fs (dm-0): recovery complete [ 1243.786865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.239097] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1249.271095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 1249.271132] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:993) [ 1250.350935] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1250.858160] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1254.182924] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 18:11:17 (1775167877) [ 1256.710598] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1257.402356] Lustre: Failing over lustre-MDT0000 [ 1257.592854] Lustre: server umount lustre-MDT0000 complete [ 1271.615792] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1271.617684] LDISKFS-fs (dm-0): recovery complete [ 1271.623287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1272.997021] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1276.930171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1276.930271] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1278.055632] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1278.602811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1281.989606] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 18:11:44 (1775167904) [ 1284.560667] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1285.220376] Lustre: Failing over lustre-MDT0000 [ 1285.373711] Lustre: server umount lustre-MDT0000 complete [ 1299.194573] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1299.195831] LDISKFS-fs (dm-0): recovery complete [ 1299.198779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.492027] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1304.574726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1304.574727] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1305.750712] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1306.276804] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1309.679950] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:12:12 (1775167932) [ 1312.156991] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1312.825074] Lustre: Failing over lustre-MDT0000 [ 1312.978484] Lustre: server umount lustre-MDT0000 complete [ 1327.136612] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1327.138640] LDISKFS-fs (dm-0): recovery complete [ 1327.140971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1327.258646] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1327.261994] Lustre: Skipped 8 previous similar messages [ 1328.546139] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1332.733233] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 1332.733233] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 1333.840489] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1334.352134] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1337.662468] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 18:12:40 (1775167960) [ 1339.295197] Lustre: 71765:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 at adminstrative request [ 1342.855280] Lustre: Failing over lustre-MDT0000 [ 1342.944576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1342.948923] Lustre: Skipped 3 previous similar messages [ 1343.060490] Lustre: server umount lustre-MDT0000 complete [ 1355.853269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.206501] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1361.401227] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 1361.401257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1362.554239] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1363.095596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1365.336560] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1368.251198] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1370.511899] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 18:13:13 (1775167993) [ 1370.881708] Lustre: 73663:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 at adminstrative request [ 1375.646943] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 18:13:18 (1775167998) [ 1378.303461] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1379.015092] Lustre: Failing over lustre-MDT0000 [ 1379.156986] Lustre: server umount lustre-MDT0000 complete [ 1392.096593] LustreError: 6495:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1392.101794] LustreError: 6495:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 261 previous similar messages [ 1393.195633] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1393.197564] LDISKFS-fs (dm-0): recovery complete [ 1393.200799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.613322] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1398.801276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 1398.801276] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 1400.028796] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1400.581634] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1403.913460] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:13:46 (1775168026) [ 1406.315772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1406.939771] Lustre: Failing over lustre-MDT0000 [ 1407.123175] Lustre: server umount lustre-MDT0000 complete [ 1421.077265] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1421.078926] LDISKFS-fs (dm-0): recovery complete [ 1421.081225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.178904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.181393] Lustre: Skipped 17 previous similar messages [ 1422.491247] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1426.428226] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1426.428246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1427.591740] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1428.134860] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1431.324360] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 18:14:14 (1775168054) [ 1433.591297] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1434.230223] Lustre: Failing over lustre-MDT0000 [ 1434.413940] Lustre: server umount lustre-MDT0000 complete [ 1448.140200] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1448.141335] LDISKFS-fs (dm-0): recovery complete [ 1448.143940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1449.406487] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1453.559281] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1453.559289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1454.535870] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1455.040968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1458.380368] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 18:14:41 (1775168081) [ 1460.700420] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1461.260071] Lustre: Failing over lustre-MDT0000 [ 1461.469837] Lustre: server umount lustre-MDT0000 complete [ 1475.547231] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1475.548416] LDISKFS-fs (dm-0): recovery complete [ 1475.551092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1476.918054] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1480.699327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1480.700869] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1482.166828] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1482.913396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1489.774684] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 18:15:12 (1775168112) [ 1495.304563] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1496.650087] Lustre: Failing over lustre-MDT0000 [ 1496.852243] Lustre: server umount lustre-MDT0000 complete [ 1517.536358] Lustre: 3643:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775168124/real 1775168124] req@ffff8ed53c357480 x1861397189273344/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1775168140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1519.408942] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1519.411180] LDISKFS-fs (dm-0): recovery complete [ 1519.424129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.773246] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x64cb67178c9e689 [ 1531.878286] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1532.399133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1532.410981] Lustre: Skipped 68 previous similar messages [ 1532.470685] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 1532.474224] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 1538.128488] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1539.331831] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1544.393498] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 18:16:07 (1775168167) [ 1548.565890] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1549.787485] Lustre: Failing over lustre-MDT0000 [ 1549.973745] Lustre: server umount lustre-MDT0000 complete [ 1566.295307] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1566.299951] LDISKFS-fs (dm-0): recovery complete [ 1566.305704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.384925] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1571.884450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1571.884547] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1573.550113] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1574.305238] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1579.303663] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:16:41 (1775168201) [ 1583.502387] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1584.700779] Lustre: Failing over lustre-MDT0000 [ 1584.867320] Lustre: server umount lustre-MDT0000 complete [ 1601.594867] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1601.598226] LDISKFS-fs (dm-0): recovery complete [ 1601.608348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.276789] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1607.215478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1607.215502] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1609.161438] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1610.317992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1615.398439] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 18:17:18 (1775168238) [ 1619.592457] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1620.632516] Lustre: Failing over lustre-MDT0000 [ 1620.814547] Lustre: server umount lustre-MDT0000 complete [ 1637.176701] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1637.179632] LDISKFS-fs (dm-0): recovery complete [ 1637.189458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1639.759819] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1642.489441] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1642.494591] Lustre: Skipped 16 previous similar messages [ 1642.519671] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1642.520962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1644.146717] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1644.913416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1649.890563] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:17:52 (1775168272) [ 1655.007211] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1656.160530] Lustre: Failing over lustre-MDT0000 [ 1656.322712] Lustre: server umount lustre-MDT0000 complete [ 1657.825055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.832192] Lustre: Skipped 65 previous similar messages [ 1673.688398] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1673.690490] LDISKFS-fs (dm-0): recovery complete [ 1673.697773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1673.766293] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1673.772549] LustreError: Skipped 16 previous similar messages [ 1675.981393] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1678.851856] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1678.860084] Lustre: Skipped 16 previous similar messages [ 1679.394972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1679.394978] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1681.484251] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1682.466249] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1688.745571] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 18:18:31 (1775168311) [ 1694.092392] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1695.372616] Lustre: Failing over lustre-MDT0000 [ 1695.520416] Lustre: server umount lustre-MDT0000 complete [ 1712.956460] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1712.958332] LDISKFS-fs (dm-0): recovery complete [ 1712.967081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.885120] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1718.300870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1718.303769] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1721.455170] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1722.599432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1728.679055] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 18:19:11 (1775168351) [ 1733.748711] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1735.012053] Lustre: Failing over lustre-MDT0000 [ 1735.203907] Lustre: server umount lustre-MDT0000 complete [ 1738.722890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1738.727448] LustreError: Skipped 15 previous similar messages [ 1753.093168] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1753.100505] LDISKFS-fs (dm-0): recovery complete [ 1753.117607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.028853] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1758.751513] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 1758.752625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 1761.179667] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1762.247849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1767.923856] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 18:19:50 (1775168390) [ 1768.592893] Lustre: 94026:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 at adminstrative request [ 1775.550754] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 18:19:58 (1775168398) [ 1779.670169] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1780.827064] Lustre: Failing over lustre-MDT0000 [ 1783.036283] Lustre: server umount lustre-MDT0000 complete [ 1790.493599] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1790.496959] LDISKFS-fs (dm-0): recovery complete [ 1790.508249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.812059] Lustre: lustre-MDT0000: Aborting client recovery [ 1790.814904] LustreError: 95307:0:(ldlm_lib.c:2983:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1790.815128] LustreError: 95344:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -108 [ 1790.817187] Lustre: 95345:0:(ldlm_lib.c:2386:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1790.837407] LustreError: 95344:0:(lod_dev.c:510:lod_sub_recovery_thread()) Skipped 1 previous similar message [ 1790.843332] Lustre: 95345:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@ [ 1790.850114] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1790.856925] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1790.877172] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1790.928580] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1505) [ 1790.941582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505) [ 1793.252593] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1796.066342] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1808.750321] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 18:20:31 (1775168431) [ 1813.351522] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1814.304529] Lustre: Failing over lustre-MDT0000 [ 1814.434556] Lustre: server umount lustre-MDT0000 complete [ 1821.576888] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1821.579353] LDISKFS-fs (dm-0): recovery complete [ 1821.586137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1821.821656] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1821.843627] Lustre: lustre-MDT0000: Aborting client recovery [ 1821.847079] LustreError: 97161:0:(ldlm_lib.c:2983:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1821.852494] Lustre: 97198:0:(ldlm_lib.c:2386:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1821.858333] Lustre: 97198:0:(ldlm_lib.c:2386:target_recovery_overseer()) Skipped 2 previous similar messages [ 1821.862953] Lustre: 97198:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1821.870695] Lustre: 97198:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1821.875123] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1821.883777] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 1821.895431] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 1821.933035] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1516 to 0x2c0000401:1537) [ 1821.933121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1516 to 0x280000401:1537) [ 1823.890892] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1827.314083] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1834.414494] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1834.419552] Lustre: Skipped 1 previous similar message [ 1837.507557] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 18:21:00 (1775168460) [ 1841.118104] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1842.112385] Lustre: Failing over lustre-MDT0000 [ 1842.239249] Lustre: server umount lustre-MDT0000 complete [ 1847.880367] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1847.882797] LDISKFS-fs (dm-0): recovery complete [ 1847.889393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.086252] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1848.086319] Lustre: lustre-MDT0000: Aborting client recovery [ 1848.089361] Lustre: Skipped 18 previous similar messages [ 1848.091809] LustreError: 99017:0:(ldlm_lib.c:2983:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1848.093569] Lustre: 99053:0:(ldlm_lib.c:2386:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1848.101480] Lustre: 99053:0:(ldlm_lib.c:2386:target_recovery_overseer()) Skipped 2 previous similar messages [ 1848.107049] Lustre: 99053:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1848.112314] Lustre: 99053:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1848.115475] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1848.120785] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 1848.137857] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 1848.181622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 1848.185105] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 1850.510071] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1853.413394] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1864.880689] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 18:21:27 (1775168487) [ 1865.442355] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1865.446243] LustreError: 6519:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed535fa0700 x1861397180856192/t201863462916(0) o36->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:419/0 lens 512/456 e 0 to 0 dl 1775168499 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1868.441380] Lustre: Failing over lustre-MDT0000 [ 1868.569815] Lustre: server umount lustre-MDT0000 complete [ 1872.798804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.013405] Lustre: lustre-MDT0000: Aborting client recovery [ 1873.016975] LustreError: 100329:0:(ldlm_lib.c:2983:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1873.023098] Lustre: 100366:0:(ldlm_lib.c:2386:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1873.028305] Lustre: 100366:0:(ldlm_lib.c:2386:target_recovery_overseer()) Skipped 2 previous similar messages [ 1873.031431] Lustre: 100366:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1873.036536] Lustre: 100366:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1873.041044] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1873.046542] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 1873.061153] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 1873.100476] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1601) [ 1873.100881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 1874.960706] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1877.991576] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1888.581903] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1889.382610] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 18:21:52 (1775168512) [ 1893.534809] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1895.195739] Lustre: Failing over lustre-MDT0000 [ 1895.379829] Lustre: server umount lustre-MDT0000 complete [ 1902.433217] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1902.436440] LDISKFS-fs (dm-0): recovery complete [ 1902.445437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1902.733145] Lustre: lustre-MDT0000: Aborting client recovery [ 1902.735586] LustreError: 102280:0:(ldlm_lib.c:2983:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1902.742474] Lustre: 102316:0:(ldlm_lib.c:2386:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1902.751939] Lustre: 102316:0:(ldlm_lib.c:2386:target_recovery_overseer()) Skipped 2 previous similar messages [ 1902.759324] Lustre: 102316:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1902.766696] Lustre: 102316:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1902.772465] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1902.779833] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 1902.788775] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 1902.825440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 1902.825940] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1633) [ 1904.896183] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1908.195976] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1919.129394] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:22:21 (1775168541) [ 1937.785289] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1938.760846] Lustre: Failing over lustre-MDT0000 [ 1938.913040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.916051] Lustre: Skipped 3 previous similar messages [ 1938.998851] Lustre: server umount lustre-MDT0000 complete [ 1955.494300] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1955.499320] LDISKFS-fs (dm-0): recovery complete [ 1955.507170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.730955] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1961.030459] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 1961.038022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 1962.854605] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1963.681132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1976.395952] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:23:19 (1775168599) [ 1989.793919] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1995.210087] Lustre: Failing over lustre-MDT0000 [ 1995.517946] Lustre: server umount lustre-MDT0000 complete [ 1996.769947] LustreError: 18180:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.781450] LustreError: 18180:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 234 previous similar messages [ 2010.485554] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2010.487017] LDISKFS-fs (dm-0): recovery complete [ 2010.492825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.198039] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2016.737343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 2016.737366] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 2018.164834] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2018.799616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2028.646480] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 18:24:11 (1775168651) [ 2029.480094] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2029.895904] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2029.902094] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2032.877167] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 18:24:15 (1775168655) [ 2045.184686] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2050.742073] Lustre: Failing over lustre-OST0000 [ 2050.806057] Lustre: server umount lustre-OST0000 complete [ 2066.559133] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 2066.562115] LDISKFS-fs (dm-2): recovery complete [ 2066.573486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2066.690214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.693742] Lustre: Skipped 16 previous similar messages [ 2069.313987] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2117.053198] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 18:25:39 (1775168739) [ 2120.551716] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2121.729357] Lustre: Failing over lustre-MDT0000 [ 2121.850894] Lustre: server umount lustre-MDT0000 complete [ 2137.072208] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2137.074594] LDISKFS-fs (dm-0): recovery complete [ 2137.081678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.944128] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2142.691382] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2142.694704] Lustre: Skipped 58 previous similar messages [ 2142.733317] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2142.734766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 2144.422938] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2145.152711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2159.072117] Lustre: 111398:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775168766/real 1775168766] req@ffff8ed53b797480 x1861397190156288/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1775168782 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 2159.092897] LustreError: 111398:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2159.094110] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2160.000569] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 18:26:22 (1775168782) [ 2160.160753] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 2162.440420] LustreError: 18180:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_race id 701 sleeping [ 2167.776202] LustreError: 18180:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2167.780892] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnecting [ 2168.603779] LustreError: 6519:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_race id 701 sleeping [ 2173.920131] LustreError: 6519:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2173.923947] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnecting [ 2174.665475] LustreError: 6490:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_race id 701 sleeping [ 2180.064123] LustreError: 6490:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2180.068139] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnecting [ 2180.974206] LustreError: 6519:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_race id 701 sleeping [ 2186.208435] LustreError: 6519:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2187.188856] LustreError: 6490:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_race id 701 sleeping [ 2192.352119] LustreError: 6490:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2192.356125] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnecting [ 2192.359892] Lustre: Skipped 1 previous similar message [ 2199.358127] LustreError: 9266:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_race id 701 sleeping [ 2199.361636] LustreError: 9266:0:(ldlm_lib.c:1164:target_handle_connect()) Skipped 1 previous similar message [ 2204.640638] LustreError: 9266:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2204.647690] LustreError: 9266:0:(ldlm_lib.c:1164:target_handle_connect()) Skipped 1 previous similar message [ 2210.784221] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnecting [ 2210.788393] Lustre: Skipped 2 previous similar messages [ 2217.731393] LustreError: 18180:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_race id 701 sleeping [ 2217.736028] LustreError: 18180:0:(ldlm_lib.c:1164:target_handle_connect()) Skipped 2 previous similar messages [ 2223.074556] LustreError: 18180:0:(ldlm_lib.c:1164:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2223.081286] LustreError: 18180:0:(ldlm_lib.c:1164:target_handle_connect()) Skipped 2 previous similar messages [ 2226.632624] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 18:27:29 (1775168849) [ 2227.436628] LustreError: 18180:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2236.933364] Lustre: lustre-MDT0000: Export ffff8ed50793d000 already connecting from 192.168.203.41@tcp [ 2242.051932] Lustre: lustre-MDT0000: Export ffff8ed50793d000 already connecting from 192.168.203.41@tcp [ 2247.171087] Lustre: lustre-MDT0000: Export ffff8ed50793d000 already connecting from 192.168.203.41@tcp [ 2252.294287] Lustre: lustre-MDT0000: Export ffff8ed50793d000 already connecting from 192.168.203.41@tcp [ 2252.299452] Lustre: Skipped 1 previous similar message [ 2257.410738] Lustre: lustre-MDT0000: Export ffff8ed50793d000 already connecting from 192.168.203.41@tcp [ 2267.480186] LustreError: 18180:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2267.484070] Lustre: 18180:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8ed541b84e00 x1861397183493632/t0(0) o38->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:0/0 lens 520/416 e 0 to 0 dl 1775168870 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2267.652457] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnecting [ 2267.658074] Lustre: Skipped 3 previous similar messages [ 2267.660872] LustreError: 6491:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2293.251238] Lustre: lustre-MDT0000: Export ffff8ed50793d000 already connecting from 192.168.203.41@tcp [ 2293.254830] Lustre: Skipped 1 previous similar message [ 2307.704123] LustreError: 6491:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2307.707719] Lustre: 6491:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff8ed535b4c000 x1861397183497856/t0(0) o38->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:0/0 lens 520/416 e 0 to 0 dl 1775168910 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2308.610986] LustreError: 6490:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2334.211566] Lustre: lustre-MDT0000: Export ffff8ed50793d000 already connecting from 192.168.203.41@tcp [ 2334.216156] Lustre: Skipped 2 previous similar messages [ 2348.656138] LustreError: 6490:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2348.662705] Lustre: 6490:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8ed52c477b80 x1861397183500928/t0(0) o38->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:0/0 lens 520/416 e 0 to 0 dl 1775168951 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2349.570750] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnecting [ 2349.575250] Lustre: Skipped 1 previous similar message [ 2349.577191] LustreError: 9266:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2375.172357] Lustre: lustre-MDT0000: Export ffff8ed50793d000 already connecting from 192.168.203.41@tcp [ 2375.180090] Lustre: Skipped 2 previous similar messages [ 2389.632140] LustreError: 9266:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2389.636510] Lustre: 9266:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8ed53c257100 x1861397183504000/t0(0) o38->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:0/0 lens 520/416 e 0 to 0 dl 1775168992 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2390.534462] LustreError: 6490:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2430.584124] LustreError: 6490:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2430.587667] Lustre: 6490:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8ed53c257480 x1861397183507200/t0(0) o38->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:0/0 lens 520/416 e 0 to 0 dl 1775169033 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2431.490319] LustreError: 6492:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2439.504152] LustreError: 6492:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 2441.325117] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 18:31:04 (1775169064) [ 2444.171929] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2445.761810] Lustre: Failing over lustre-MDT0000 [ 2445.793604] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2445.793908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2445.800187] Lustre: Skipped 49 previous similar messages [ 2445.802626] Lustre: Skipped 1 previous similar message [ 2445.878982] Lustre: server umount lustre-MDT0000 complete [ 2450.804144] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2450.806947] LDISKFS-fs (dm-0): recovery complete [ 2450.814719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.881073] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2450.884402] LustreError: Skipped 10 previous similar messages [ 2450.913636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2450.918061] Lustre: Skipped 5 previous similar messages [ 2450.966507] Lustre: *** cfs_fail_loc=712, val=0*** [ 2450.969957] LustreError: 8672:0:(service.c:1390:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8ed53c257480 x1861397190302720/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2451.041233] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2451.041507] Lustre: lustre-MDT0000: Aborting client recovery [ 2451.045337] Lustre: Skipped 12 previous similar messages [ 2451.049710] LustreError: 116144:0:(ldlm_lib.c:2983:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2451.053589] Lustre: 116180:0:(ldlm_lib.c:2386:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2451.057751] Lustre: 116180:0:(ldlm_lib.c:2386:target_recovery_overseer()) Skipped 2 previous similar messages [ 2451.061565] Lustre: 116180:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2451.066013] Lustre: 116180:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2451.070264] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2451.075849] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2451.084474] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2451.115286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2451.115402] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2452.756514] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2456.033882] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2463.212459] Lustre: Failing over lustre-MDT0000 [ 2463.374320] Lustre: server umount lustre-MDT0000 complete [ 2476.645630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.571716] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2477.575920] Lustre: Skipped 6 previous similar messages [ 2478.350138] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2482.155499] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2482.159987] Lustre: Skipped 7 previous similar messages [ 2482.179463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2482.179482] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2483.562450] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2484.211030] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2487.923528] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 18:31:50 (1775169110) [ 2487.972518] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnecting [ 2487.976446] Lustre: Skipped 2 previous similar messages [ 2491.503719] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 18:31:54 (1775169114) [ 2491.876345] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2491.878497] LustreError: 6500:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed53dcff100 x1861397183563648/t0(0) o700->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:291/0 lens 264/248 e 0 to 0 dl 1775169126 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2509.575576] Lustre: Failing over lustre-MDT0000 [ 2509.794737] Lustre: server umount lustre-MDT0000 complete [ 2512.866559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2512.870629] LustreError: Skipped 6 previous similar messages [ 2523.157860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2524.892433] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2528.770944] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2528.770958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2530.156887] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2530.809550] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2535.322988] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:32:38 (1775169158) [ 2536.369718] Lustre: Failing over lustre-OST0000 [ 2536.435565] Lustre: server umount lustre-OST0000 complete [ 2549.524746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2551.788373] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2555.263636] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2555.890965] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2621.582340] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:34:04 (1775169244) [ 2624.430899] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2625.303817] Lustre: Failing over lustre-MDT0000 [ 2625.487187] Lustre: server umount lustre-MDT0000 complete [ 2626.017079] LustreError: 7504:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.025408] LustreError: 7504:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 73 previous similar messages [ 2639.940836] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2639.942822] LDISKFS-fs (dm-0): recovery complete [ 2639.947648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.532952] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2645.623138] Lustre: *** cfs_fail_loc=216, val=0*** [ 2645.623172] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2645.625477] LustreError: 121969:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2646.688562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2708.328468] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 18:35:31 (1775169331) [ 2716.577519] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 18:35:39 (1775169339) [ 2717.671991] Lustre: Failing over lustre-MDT0000 [ 2717.893666] Lustre: server umount lustre-MDT0000 complete [ 2730.907038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2731.036285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.039085] Lustre: Skipped 6 previous similar messages [ 2732.407675] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2736.101206] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2736.103511] LustreError: 123544:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed531153b80 x1861397183680896/t0(0) o101->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:577/0 lens 328/344 e 0 to 0 dl 1775169412 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2795.021892] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnected, waiting for 2 clients in recovery for 1:05 [ 2795.029902] Lustre: 123544:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2795.038260] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2795.041108] Lustre: Skipped 27 previous similar messages [ 2795.053885] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 2795.053912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 2796.378123] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2796.934343] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2800.804234] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 18:37:03 (1775169423) [ 2802.189669] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2805.055417] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2805.736481] Lustre: Failing over lustre-MDT0000 [ 2805.874825] Lustre: server umount lustre-MDT0000 complete [ 2820.324869] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2820.326534] LDISKFS-fs (dm-0): recovery complete [ 2820.329123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2821.792993] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2825.718526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 2825.718564] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 2826.932858] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2827.479460] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2830.945525] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 18:37:33 (1775169453) [ 2831.326214] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2835.142858] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2835.807160] Lustre: Failing over lustre-MDT0000 [ 2835.898210] Lustre: server umount lustre-MDT0000 complete [ 2850.039487] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2850.040792] LDISKFS-fs (dm-0): recovery complete [ 2850.044263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.407543] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2855.411378] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3137) [ 2855.411418] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 2856.563890] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2857.114834] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2860.638455] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 18:38:03 (1775169483) [ 2861.012268] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2864.812616] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2865.428033] Lustre: Failing over lustre-MDT0000 [ 2865.511313] Lustre: server umount lustre-MDT0000 complete [ 2879.616659] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2879.618395] LDISKFS-fs (dm-0): recovery complete [ 2879.621560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.022567] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2885.109355] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 2885.109384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3201) [ 2889.351274] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 18:38:32 (1775169512) [ 2890.718786] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2890.720096] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2890.722550] LustreError: 6493:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed502906450 x1861397183731072/t257698037777(0) o35->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:690/0 lens 392/456 e 0 to 0 dl 1775169525 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2891.618747] Lustre: Failing over lustre-MDT0000 [ 2891.803421] Lustre: server umount lustre-MDT0000 complete [ 2904.657171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.074296] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2910.190901] Lustre: 38455:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed53c625500 x1861397183731072/t257698037777(0) o35->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:709/0 lens 392/456 e 0 to 0 dl 1775169544 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2910.198151] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3201) [ 2910.198207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 2911.345587] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2911.913296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2915.268634] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 18:38:58 (1775169538) [ 2915.631737] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2915.633774] LustreError: 6491:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed409760700 x1861397183743872/t261993005072(0) o36->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:3/0 lens 504/448 e 0 to 0 dl 1775169593 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2919.376503] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2920.019069] Lustre: Failing over lustre-MDT0000 [ 2920.096374] Lustre: server umount lustre-MDT0000 complete [ 2934.015576] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2934.016965] LDISKFS-fs (dm-0): recovery complete [ 2934.019877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2935.429875] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2939.372864] Lustre: 6491:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed405f80e00 x1861397183743872/t261993005072(0) o36->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:27/0 lens 504/2880 e 0 to 0 dl 1775169617 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2939.379175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3265) [ 2939.379177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 2940.540214] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2941.086624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2944.567549] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 18:39:27 (1775169567) [ 2944.939907] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2944.941529] LustreError: 18180:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed53c60b480 x1861397183757696/t266287972368(0) o36->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:33/0 lens 504/448 e 0 to 0 dl 1775169623 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2946.212898] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2948.574994] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2949.193872] Lustre: Failing over lustre-MDT0000 [ 2949.273988] Lustre: server umount lustre-MDT0000 complete [ 2963.288103] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2963.289396] LDISKFS-fs (dm-0): recovery complete [ 2963.292384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.594098] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2968.554393] Lustre: 38455:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed535b4d880 x1861397183757952/t266287972369(0) o35->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:12/0 lens 392/456 e 0 to 0 dl 1775169602 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2968.560673] Lustre: 38455:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 2968.561895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3297) [ 2968.561896] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 2972.632660] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 18:39:55 (1775169595) [ 2972.953728] LustreError: 9266:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed409954e00 x1861397183769856/t270582939664(0) o36->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:61/0 lens 504/448 e 0 to 0 dl 1775169651 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2972.960070] LustreError: 9266:0:(ldlm_lib.c:3325:target_send_reply_msg()) Skipped 1 previous similar message [ 2974.236228] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2974.238832] Lustre: Skipped 1 previous similar message [ 2977.052502] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2977.655741] Lustre: Failing over lustre-MDT0000 [ 2977.733852] Lustre: server umount lustre-MDT0000 complete [ 2991.934880] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2991.936364] LDISKFS-fs (dm-0): recovery complete [ 2991.939225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.321573] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2997.229959] Lustre: 6492:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed541b87800 x1861397183769856/t270582939664(0) o36->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:85/0 lens 504/2880 e 0 to 0 dl 1775169675 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2997.237916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 2997.237941] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3297) [ 3001.498472] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 18:40:24 (1775169624) [ 3001.905336] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3003.224209] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3003.225847] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 3003.228050] Lustre: Skipped 2 previous similar messages [ 3003.230394] LustreError: 38455:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed531151180 x1861397183782272/t274877906960(0) o35->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:47/0 lens 392/456 e 0 to 0 dl 1775169637 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3006.855745] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3007.521996] Lustre: Failing over lustre-MDT0000 [ 3007.607588] Lustre: server umount lustre-MDT0000 complete [ 3021.873092] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3021.875020] LDISKFS-fs (dm-0): recovery complete [ 3021.879120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3023.351692] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3027.438649] Lustre: 6494:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed53164c380 x1861397183782272/t274877906960(0) o35->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:71/0 lens 392/456 e 0 to 0 dl 1775169661 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3027.444287] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3329) [ 3027.444330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 3031.850813] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 18:40:54 (1775169654) [ 3032.176746] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 3091.976450] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnecting [ 3091.980171] Lustre: Skipped 4 previous similar messages [ 3091.987551] Lustre: 6490:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed52c747480 x1861397183792640/t279172874255(0) o101->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:180/0 lens 664/3488 e 0 to 0 dl 1775169770 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 3094.250697] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 18:41:57 (1775169717) [ 3096.769771] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3097.399583] Lustre: Failing over lustre-MDT0000 [ 3097.484409] Lustre: server umount lustre-MDT0000 complete [ 3099.104772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.108879] Lustre: Skipped 55 previous similar messages [ 3111.251854] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3111.253109] LDISKFS-fs (dm-0): recovery complete [ 3111.255991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.287432] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3111.291392] LustreError: Skipped 12 previous similar messages [ 3111.390243] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3111.392308] Lustre: Skipped 15 previous similar messages [ 3112.450547] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3112.453814] Lustre: Skipped 12 previous similar messages [ 3112.654315] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3116.524235] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3116.526570] Lustre: Skipped 12 previous similar messages [ 3116.541257] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3361) [ 3116.541261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 3117.616294] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3118.143977] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3131.475465] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 18:42:34 (1775169754) [ 3134.069741] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3134.663310] Lustre: Failing over lustre-MDT0000 [ 3134.820053] Lustre: server umount lustre-MDT0000 complete [ 3136.993015] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3136.995655] LustreError: Skipped 8 previous similar messages [ 3148.621740] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3148.622978] LDISKFS-fs (dm-0): recovery complete [ 3148.625479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3149.932835] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3153.915380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 3153.915380] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3393) [ 3154.991626] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3155.492840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3157.747416] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3162.614729] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 18:43:05 (1775169785) [ 3170.791820] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3171.354270] Lustre: Failing over lustre-MDT0000 [ 3171.576749] Lustre: server umount lustre-MDT0000 complete [ 3185.579104] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3185.580703] LDISKFS-fs (dm-0): recovery complete [ 3185.583717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.946646] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3190.863990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 3190.863990] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 3191.949981] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3192.458198] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3205.642204] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 18:43:48 (1775169828) [ 3209.021225] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3209.609017] Lustre: Failing over lustre-MDT0000 [ 3209.754164] Lustre: server umount lustre-MDT0000 complete [ 3223.599055] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3223.600936] LDISKFS-fs (dm-0): recovery complete [ 3223.603848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3224.919067] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3229.185144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 3229.185149] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 3230.346599] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3230.903467] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3233.866919] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3234.433951] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3236.569959] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 18:44:19 (1775169859) [ 3242.387072] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3302.624177] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3302.625497] Lustre: Skipped 1 previous similar message [ 3302.626830] LustreError: 6490:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed50805fc50 x1861397186579712/t296352743435(0) o36->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:390/0 lens 66040/440 e 0 to 0 dl 1775169980 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3302.634545] LustreError: 6490:0:(ldlm_lib.c:3325:target_send_reply_msg()) Skipped 1 previous similar message [ 3363.331843] Lustre: 6519:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed406e5c700 x1861397186579712/t296352743435(0) o36->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:451/0 lens 66040/440 e 0 to 0 dl 1775170041 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3366.207666] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3366.706176] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 18:46:29 (1775169989) [ 3370.563590] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3371.317757] Lustre: Failing over lustre-MDT0000 [ 3371.466582] Lustre: server umount lustre-MDT0000 complete [ 3372.512774] LustreError: 6496:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.517974] LustreError: 6496:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 200 previous similar messages [ 3385.250861] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3385.251995] LDISKFS-fs (dm-0): recovery complete [ 3385.255023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.347466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.350137] Lustre: Skipped 12 previous similar messages [ 3386.533453] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3390.577664] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 3390.577815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 3391.643825] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3392.166154] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3395.861827] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 18:46:58 (1775170018) [ 3401.055407] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3403.908222] Lustre: Failing over lustre-OST0000 [ 3403.945809] Lustre: server umount lustre-OST0000 complete [ 3417.713026] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3417.714548] LDISKFS-fs (dm-2): recovery complete [ 3417.717690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3419.536608] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3419.757762] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3419.760256] Lustre: Skipped 52 previous similar messages [ 3430.701399] Lustre: Failing over lustre-OST0000 [ 3430.740087] Lustre: server umount lustre-OST0000 complete [ 3443.371043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3445.111018] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3448.117946] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3448.663417] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3482.306263] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 18:48:25 (1775170105) [ 3483.149038] Lustre: Failing over lustre-MDT0000 [ 3483.376850] Lustre: server umount lustre-MDT0000 complete [ 3495.975818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.246475] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3501.555444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 3501.555444] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 3508.421882] Lustre: Failing over lustre-MDT0000 [ 3508.569196] Lustre: server umount lustre-MDT0000 complete [ 3521.180671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3522.475434] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3526.643421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 3526.643484] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 3527.696461] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3528.194663] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3531.565371] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 18:49:14 (1775170154) [ 3542.693605] Lustre: Failing over lustre-OST0000 [ 3542.727452] Lustre: server umount lustre-OST0000 complete [ 3555.412689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3557.158687] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3560.012814] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3560.578117] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3564.234830] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 18:49:47 (1775170187) [ 3564.805679] Lustre: Failing over lustre-MDT0000 [ 3565.000628] Lustre: server umount lustre-MDT0000 complete [ 3567.892100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3567.923199] Lustre: *** cfs_fail_loc=605, val=0*** [ 3567.924592] LustreError: 157284:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc08fc2d0 failed: rc = -95 [ 3567.927502] LustreError: 157284:0:(obd_config.c:843:class_setup()) setup MGS failed (-95) [ 3567.929536] LustreError: 157284:0:(obd_mount.c:250:lustre_start_simple()) MGS setup error -95 [ 3567.931679] LustreError: 157284:0:(tgt_mount.c:114:server_deregister_mount()) MGS not registered [ 3567.933623] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3567.935214] LustreError: 157284:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 3567.939044] Lustre: server umount lustre-MDT0000 complete [ 3567.940418] LustreError: 157284:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3570.302133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3571.545986] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3573.973940] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 18:49:57 (1775170197) [ 3575.799212] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3575.799232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3578.189276] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3579.101869] Lustre: Failing over lustre-MDT0000 [ 3579.300948] Lustre: server umount lustre-MDT0000 complete [ 3593.065901] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3593.067642] LDISKFS-fs (dm-0): recovery complete [ 3593.070684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.731660] Lustre: *** cfs_fail_loc=707, val=0*** [ 3594.412791] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3654.163589] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 3654.175270] Lustre: 159447:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3654.179107] Lustre: 159447:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 3654.277637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 3654.277637] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 3655.550553] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3656.058443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3659.669530] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 18:51:22 (1775170282) [ 3683.073395] LustreError: 6492:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8ed53586b800 x1861397187456384/t0(0) o101->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:727/0 lens 664/0 e 0 to 0 dl 1775170317 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3683.079970] LustreError: 6492:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3694.104102] LustreError: 6492:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3694.110219] LustreError: 6493:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8ed53c355f80 x1861397187457664/t0(0) o35->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:738/0 lens 392/0 e 0 to 0 dl 1775170328 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3700.193890] LustreError: 15175:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8ed53c35ad80 x1861397191902208/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:744/0 lens 544/0 e 0 to 0 dl 1775170334 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3700.203908] LustreError: 15175:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 48 previous similar messages [ 3705.972149] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 18:52:08 (1775170328) [ 3729.354858] LustreError: 41082:0:(tgt_handler.c:2787:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3740.384061] LustreError: 41082:0:(tgt_handler.c:2787:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3743.036454] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 18:52:46 (1775170366) [ 3765.891684] LustreError: 9266:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8ed53c149c00 x1861397187481472/t0(0) o101->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:55/0 lens 576/0 e 0 to 0 dl 1775170400 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3765.904381] LustreError: 9266:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3771.000122] LustreError: 9266:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3771.005685] LustreError: 6491:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8ed53c355180 x1861397187482368/t0(0) o101->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:60/0 lens 664/0 e 0 to 0 dl 1775170405 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3781.832054] LustreError: 6492:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3781.836490] LustreError: 18180:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8ed53cad9500 x1861397187504128/t0(0) o101->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:110/0 lens 664/0 e 0 to 0 dl 1775170455 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3781.843475] LustreError: 18180:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 3793.704233] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 18:53:36 (1775170416) [ 3878.713183] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 18:55:01 (1775170501) [ 3901.648989] LustreError: 6519:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8ed53c60ce00 x1861397187561344/t0(0) o101->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:190/0 lens 576/0 e 0 to 0 dl 1775170535 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3901.659064] LustreError: 6519:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 3901.662933] LustreError: 6519:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3901.665198] LustreError: 6519:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 3902.080065] LustreError: 6519:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3917.897740] LustreError: 6519:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3917.900769] LustreError: 6519:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3918.320066] LustreError: 6519:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3918.324353] LustreError: 6519:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3933.730295] LustreError: 6519:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff8ed4086fa300 x1861397187580800/t0(0) o36->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:223/0 lens 504/0 e 0 to 0 dl 1775170568 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 3933.740502] LustreError: 6519:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 81 previous similar messages [ 3945.947565] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 18:56:08 (1775170568) [ 3970.428315] Lustre: DEBUG MARKER: phase 2 [ 3973.403209] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 18:56:36 (1775170596) [ 4043.342520] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 18:57:46 (1775170666) [ 4043.810321] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 4044.363181] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 18:57:47 (1775170667) [ 4046.116866] Lustre: DEBUG MARKER: Started rundbench load pid=134191 ... [ 4049.813671] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4051.335978] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 4051.998836] Lustre: Failing over lustre-MDT0000 [ 4052.013974] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [ 4052.206445] Lustre: server umount lustre-MDT0000 complete [ 4053.984843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.984857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4053.985221] LustreError: 6492:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4053.985228] LustreError: 6492:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 75 previous similar messages [ 4053.988599] Lustre: Skipped 42 previous similar messages [ 4053.992028] LustreError: Skipped 6 previous similar messages [ 4066.140941] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4066.142273] LDISKFS-fs (dm-0): recovery complete [ 4066.145611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.180415] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4066.183868] LustreError: Skipped 8 previous similar messages [ 4066.255355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4066.257377] Lustre: Skipped 7 previous similar messages [ 4066.272519] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4066.275210] Lustre: Skipped 11 previous similar messages [ 4067.512540] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4069.890231] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4069.894605] Lustre: Skipped 11 previous similar messages [ 4071.394094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4071.396097] Lustre: Skipped 21 previous similar messages [ 4071.752214] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4071.755152] Lustre: Skipped 11 previous similar messages [ 4071.773051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5134 to 0x280000401:5153) [ 4071.773083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5098 to 0x2c0000401:5121) [ 4072.909734] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4073.485282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4078.361380] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4079.875131] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 4080.518938] Lustre: Failing over lustre-MDT0001 [ 4080.688525] Lustre: server umount lustre-MDT0001 complete [ 4095.024116] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 4095.025459] LDISKFS-fs (dm-1): recovery complete [ 4095.031138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4096.575249] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4100.599443] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) [ 4100.599717] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:449) [ 4101.862166] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4102.479275] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4107.508897] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4109.014385] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 4109.645469] Lustre: Failing over lustre-MDT0000 [ 4109.649912] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [ 4109.861803] Lustre: server umount lustre-MDT0000 complete [ 4123.761405] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4123.762662] LDISKFS-fs (dm-0): recovery complete [ 4123.766060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4125.088047] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4129.492464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5558 to 0x280000401:5601) [ 4129.492481] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5525 to 0x2c0000401:5569) [ 4130.655442] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4131.192082] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4136.229377] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4137.758252] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 4138.419738] Lustre: Failing over lustre-MDT0001 [ 4138.571521] Lustre: server umount lustre-MDT0001 complete [ 4152.598725] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 4152.600154] LDISKFS-fs (dm-1): recovery complete [ 4152.604023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4154.041031] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4157.946779] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:481) [ 4157.946792] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:481) [ 4159.135141] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4159.677693] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4164.557520] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4166.072150] Lustre: DEBUG MARKER: test_70b fail mds1 5 times [ 4166.718833] Lustre: Failing over lustre-MDT0000 [ 4166.723244] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [ 4166.794574] LustreError: 6490:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000009fc5127b ns: mdt-lustre-MDT0000_UUID lock: ffff8ed53c490200/0x64cb67178de4aa6 lrc: 3/0,0 mode: PR/PR res: [0x20001a9e3:0x1297:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.203.41@tcp remote: 0x49029cbec65dfb07 expref: 3 pid: 6490 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4166.971738] Lustre: server umount lustre-MDT0000 complete [ 4181.051030] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4181.052303] LDISKFS-fs (dm-0): recovery complete [ 4181.054787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4182.458500] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4186.964153] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5976 to 0x2c0000401:6017) [ 4186.964186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6009 to 0x280000401:6049) [ 4188.204205] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4188.822655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4192.223370] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 19:00:15 (1775170815) [ 4315.277714] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4326.156890] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 4326.799109] Lustre: Failing over lustre-MDT0000 [ 4326.812638] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [ 4327.091502] Lustre: server umount lustre-MDT0000 complete [ 4341.156927] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4341.158298] LDISKFS-fs (dm-0): recovery complete [ 4341.162320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4342.603321] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4348.557417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7472 to 0x280000401:7489) [ 4348.559953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7439 to 0x2c0000401:7457) [ 4349.814861] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4350.347645] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4474.509603] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4485.335328] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 4485.954644] Lustre: Failing over lustre-MDT0000 [ 4485.991106] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [ 4491.870242] Lustre: server umount lustre-MDT0000 complete [ 4505.996202] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4505.997674] LDISKFS-fs (dm-0): recovery complete [ 4506.001075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4507.431812] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4513.130040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9077 to 0x280000401:9121) [ 4513.130368] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9046 to 0x2c0000401:9089) [ 4514.187891] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4514.695345] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4534.231759] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 19:05:57 (1775171157) [ 4657.087672] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4667.814231] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 4668.449908] Lustre: Failing over lustre-MDT0001 [ 4668.467267] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.41@tcp (stopping) [ 4668.469268] Lustre: Skipped 7 previous similar messages [ 4668.705443] Lustre: server umount lustre-MDT0001 complete [ 4669.920856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.921229] LustreError: 6519:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4669.924277] Lustre: Skipped 24 previous similar messages [ 4669.931489] LustreError: 6519:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 97 previous similar messages [ 4669.932671] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4669.938048] LustreError: Skipped 5 previous similar messages [ 4682.615459] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4682.616788] LDISKFS-fs (dm-1): recovery complete [ 4682.620517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4682.722860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4682.727242] Lustre: Skipped 6 previous similar messages [ 4682.737283] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 4682.740451] Lustre: Skipped 6 previous similar messages [ 4683.962399] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4683.980716] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4683.983198] Lustre: Skipped 6 previous similar messages [ 4687.842073] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4687.844758] Lustre: Skipped 25 previous similar messages [ 4688.803668] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4688.807334] Lustre: Skipped 6 previous similar messages [ 4688.828717] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3926 to 0x280000400:3969) [ 4688.829547] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3926 to 0x2c0000400:3969) [ 4690.036466] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4690.632452] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4814.527921] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4825.154306] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 4825.730928] Lustre: Failing over lustre-MDT0000 [ 4825.739665] LustreError: 180413:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-MDT0001-osp-MDT0000: namespace resource [0x20001a9e3:0x6780:0x0].0x0 (ffff8ed5121b8400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4825.748275] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [ 4831.826811] Lustre: server umount lustre-MDT0000 complete [ 4845.711279] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4845.712486] LDISKFS-fs (dm-0): recovery complete [ 4845.715943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.754574] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4845.758466] LustreError: Skipped 4 previous similar messages [ 4847.126268] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4852.329392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10177 to 0x280000401:10209) [ 4852.331247] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10146 to 0x2c0000401:10177) [ 4853.605290] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4854.183897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4857.248244] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 19:11:20 (1775171480) [ 4980.410203] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4991.181476] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 4991.797243] Lustre: Failing over lustre-MDT0001 [ 4991.811354] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4991.812996] Lustre: Skipped 9 previous similar messages [ 4992.009812] Lustre: server umount lustre-MDT0001 complete [ 5005.907825] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5005.909751] LDISKFS-fs (dm-1): recovery complete [ 5005.914745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5007.368166] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5011.445955] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4436 to 0x2c0000400:4481) [ 5011.445972] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4436 to 0x280000400:4481) [ 5012.828690] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5013.556812] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5137.609603] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5148.396217] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 5149.070052] Lustre: Failing over lustre-MDT0001 [ 5149.078776] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5149.268689] Lustre: server umount lustre-MDT0001 complete [ 5163.284049] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5163.285445] LDISKFS-fs (dm-1): recovery complete [ 5163.289903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5164.615549] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5168.632617] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4436 to 0x2c0000400:4513) [ 5168.632618] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4436 to 0x280000400:4513) [ 5169.851877] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5170.552256] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5174.213335] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 19:16:37 (1775171797) [ 5180.083139] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5181.556032] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5182.168870] Lustre: Failing over lustre-OST0000 [ 5182.210392] Lustre: server umount lustre-OST0000 complete [ 5196.008915] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5196.010865] LDISKFS-fs (dm-2): recovery complete [ 5196.014632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5197.890248] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5201.060209] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5201.659441] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5210.395626] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5211.914409] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5212.519082] Lustre: Failing over lustre-OST0000 [ 5212.556725] Lustre: server umount lustre-OST0000 complete [ 5226.492398] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5226.493681] LDISKFS-fs (dm-2): recovery complete [ 5226.496393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5228.295153] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5231.503091] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5232.113252] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5237.476142] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 19:17:40 (1775171860) [ 5360.397587] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5363.668839] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5374.404654] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 5375.032154] Lustre: Failing over lustre-MDT0000 [ 5375.048455] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [ 5375.216197] Lustre: server umount lustre-MDT0000 complete [ 5376.442138] LustreError: 7415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775171999 with bad export cookie 453938260938026021 [ 5376.444080] Lustre: Failing over lustre-MDT0001 [ 5376.447259] LustreError: 7415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5376.603963] Lustre: server umount lustre-MDT0001 complete [ 5391.040346] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5391.044205] LDISKFS-fs (dm-0): recovery complete [ 5391.051171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5391.115402] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 5391.116726] LDISKFS-fs (dm-1): recovery complete [ 5391.120530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.936120] Lustre: 3642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775172003/real 1775172003] req@ffff8ed4101b1f80 x1861397236989952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775172019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5395.936188] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5395.943544] Lustre: 3642:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5395.952663] Lustre: Skipped 17 previous similar messages [ 5396.960131] Lustre: 3642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775172004/real 1775172004] req@ffff8ed40b66f100 x1861397236990464/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775172020 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5396.967774] Lustre: 3642:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5401.056439] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ed4101b1500 x1861397236992640/t0(0) o250->MGC192.168.203.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5401.154065] LustreError: 174386:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5401.164579] LustreError: 174386:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 60 previous similar messages [ 5401.179928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5401.182646] Lustre: Skipped 5 previous similar messages [ 5401.194351] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5401.198442] Lustre: Skipped 5 previous similar messages [ 5401.273082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 5401.277590] LustreError: Skipped 6 previous similar messages [ 5402.208066] Lustre: 3641:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775172009/real 1775172009] req@ffff8ed4101b0700 x1861397236990976/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775172025 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5402.215201] Lustre: 3641:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5402.585799] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5402.725280] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5406.146122] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5406.149853] Lustre: Skipped 5 previous similar messages [ 5406.434023] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5406.436991] Lustre: Skipped 16 previous similar messages [ 5407.456142] Lustre: 3641:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775172014/real 1775172014] req@ffff8ed408f2f800 x1861397236991488/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775172030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5407.463044] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5407.464675] Lustre: 3641:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 5407.466946] Lustre: Skipped 5 previous similar messages [ 5407.480490] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4436 to 0x2c0000400:4545) [ 5407.480495] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4436 to 0x280000400:4545) [ 5412.832178] Lustre: 3641:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775172019/real 1775172019] req@ffff8ed53782e300 x1861397236992128/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775172035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5419.633160] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14481 to 0x280000401:14497) [ 5419.634956] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14465) [ 5421.034928] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5421.579975] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5422.109868] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5425.350698] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 19:20:48 (1775172048) [ 5427.736308] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5428.614699] Lustre: Failing over lustre-MDT0000 [ 5428.749055] Lustre: server umount lustre-MDT0000 complete [ 5442.643863] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5442.645182] LDISKFS-fs (dm-0): recovery complete [ 5442.648665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5444.249628] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5446.148705] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5462.540926] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5462.554284] Lustre: 191378:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed41065e680 x1861397284124928/t343597427364(343597427364) o101->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:241/0 lens 520/3488 e 0 to 0 dl 1775172096 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5462.592460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14499 to 0x280000401:14529) [ 5462.592463] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14497) [ 5463.966189] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5464.497398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5467.769718] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 19:21:30 (1775172090) [ 5470.295435] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5471.170214] Lustre: Failing over lustre-MDT0000 [ 5471.402887] Lustre: server umount lustre-MDT0000 complete [ 5485.272695] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5485.274617] LDISKFS-fs (dm-0): recovery complete [ 5485.278099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.312611] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5485.316967] LustreError: Skipped 2 previous similar messages [ 5486.573266] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5487.113602] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5487.115277] LustreError: 191378:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed408b48380 x1861397284124928/t343597427364(343597427364) o101->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:266/0 lens 520/664 e 0 to 0 dl 1775172121 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5503.496504] Lustre: lustre-MDT0000: Client a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3 (at 192.168.203.41@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 5503.500609] Lustre: 192346:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed531714700 x1861397284124928/t343597427364(343597427364) o101->a1fd9bf4-1ed2-45e3-bfe9-fa2a9f9e73a3@192.168.203.41@tcp:282/0 lens 520/3488 e 0 to 0 dl 1775172137 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5503.534659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14531 to 0x280000401:14561) [ 5503.534687] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14529) [ 5504.801719] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5505.305571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5508.583633] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 19:22:11 (1775172131) [ 5510.136365] Lustre: Failing over lustre-OST0000 [ 5510.180370] Lustre: server umount lustre-OST0000 complete [ 5511.433913] Lustre: Failing over lustre-MDT0000 [ 5511.550412] Lustre: server umount lustre-MDT0000 complete [ 5524.615891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.957001] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5526.646561] Lustre: lustre-MDT0000: Denying connection for new client 1f1b116f-0937-4906-afca-85d7a7b8a129 (at 192.168.203.41@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5530.101522] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14561) [ 5533.927895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5535.153181] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14531 to 0x280000401:14593) [ 5535.719220] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5539.572342] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 19:22:42 (1775172162) [ 5539.905199] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5539.906757] LustreError: 191384:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed405f81880 x1861397238665472/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:319/0 lens 1312/4320 e 0 to 0 dl 1775172174 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5542.165023] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5542.729479] Lustre: Failing over lustre-MDT0000 [ 5542.926050] Lustre: server umount lustre-MDT0000 complete [ 5556.192122] Lustre: 191436:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775172163/real 1775172163] req@ffff8ed405f83800 x1861397238665472/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1775172179 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5556.711030] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5556.712207] LDISKFS-fs (dm-0): recovery complete [ 5556.715510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.943222] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5561.845144] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14593) [ 5561.845166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14625) [ 5562.907634] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5563.380325] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5566.714900] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 19:23:09 (1775172189) [ 5567.041413] LustreError: 194088:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed53633a300 x1861397238694272/t365072220172(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:346/0 lens 2520/4320 e 0 to 0 dl 1775172201 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5569.355166] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5571.636857] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5572.209564] Lustre: Failing over lustre-MDT0001 [ 5578.295273] Lustre: server umount lustre-MDT0001 complete [ 5592.072962] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5592.074391] LDISKFS-fs (dm-1): recovery complete [ 5592.078801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5593.409582] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5597.679252] Lustre: 191379:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed53da56300 x1861397291211904/t30064773616(0) o36->1f1b116f-0937-4906-afca-85d7a7b8a129@192.168.203.41@tcp:397/0 lens 560/2880 e 0 to 0 dl 1775172252 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5597.684443] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4556 to 0x280000400:4577) [ 5597.684494] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4556 to 0x2c0000400:4577) [ 5598.795537] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5599.300639] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5602.726758] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 19:23:45 (1775172225) [ 5603.063457] LustreError: 191383:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed53d0d0a80 x1861397238726912/t365072220193(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:382/0 lens 2520/4320 e 0 to 0 dl 1775172237 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5605.371813] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5607.752525] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5608.375149] Lustre: Failing over lustre-MDT0000 [ 5608.612048] Lustre: server umount lustre-MDT0000 complete [ 5619.680084] Lustre: 201791:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775172226/real 1775172226] req@ffff8ed53d0d3b80 x1861397238726912/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1775172242 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5622.496417] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5622.498096] LDISKFS-fs (dm-0): recovery complete [ 5622.501438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.822093] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5627.927435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14657) [ 5627.927449] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14625) [ 5628.970575] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5629.456515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5631.334857] Lustre: Failing over lustre-MDT0001 [ 5631.424413] Lustre: server umount lustre-MDT0001 complete [ 5645.266048] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5645.268056] LDISKFS-fs (dm-1): recovery complete [ 5645.274031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5646.645711] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5650.935412] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4588 to 0x280000400:4609) [ 5650.935425] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4588 to 0x2c0000400:4609) [ 5652.017898] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5652.541635] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5655.995682] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 19:24:39 (1775172279) [ 5656.334929] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5656.337504] Lustre: Skipped 2 previous similar messages [ 5661.699163] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5664.145031] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5664.757134] Lustre: Failing over lustre-MDT0000 [ 5665.019356] Lustre: server umount lustre-MDT0000 complete [ 5666.194979] LustreError: 9446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775172289 with bad export cookie 453938260941277458 [ 5666.197267] Lustre: Failing over lustre-MDT0001 [ 5666.201947] LustreError: 9446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5666.209569] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.41@tcp (stopping) [ 5666.213605] Lustre: Skipped 6 previous similar messages [ 5671.904843] LustreError: 207100:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8ed4102e1c00 x1861397238789760/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5671.914747] LustreError: 207100:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5672.048736] Lustre: server umount lustre-MDT0001 complete [ 5686.907493] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5686.909863] LDISKFS-fs (dm-0): recovery complete [ 5686.913506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5686.915992] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5686.919433] LDISKFS-fs (dm-1): recovery complete [ 5686.924522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5690.848348] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ed40ba8c700 x1861397238790528/t0(0) o250->MGC192.168.203.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5692.315935] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5692.402277] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5697.269399] Lustre: 208215:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed408f90000 x1861397291279360/t38654705732(0) o36->1f1b116f-0937-4906-afca-85d7a7b8a129@192.168.203.41@tcp:502/0 lens 560/2880 e 0 to 0 dl 1775172357 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5697.276124] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4620 to 0x280000400:4641) [ 5697.276142] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4620 to 0x2c0000400:4641) [ 5707.295922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14689) [ 5707.295922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14657) [ 5708.621783] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5709.147533] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5709.628302] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5713.022652] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 19:25:36 (1775172336) [ 5713.391973] LustreError: 208214:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed405256a00 x1861397291312128/t42949673028(0) o36->1f1b116f-0937-4906-afca-85d7a7b8a129@192.168.203.41@tcp:492/0 lens 560/448 e 0 to 0 dl 1775172347 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5713.402828] LustreError: 208214:0:(ldlm_lib.c:3325:target_send_reply_msg()) Skipped 1 previous similar message [ 5718.755873] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5719.352038] Lustre: Failing over lustre-MDT0000 [ 5719.560726] Lustre: server umount lustre-MDT0000 complete [ 5728.770180] Lustre: lustre-MDT0001: Client 1f1b116f-0937-4906-afca-85d7a7b8a129 (at 192.168.203.41@tcp) reconnecting [ 5728.772743] Lustre: Skipped 2 previous similar messages [ 5733.469669] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5733.471836] LDISKFS-fs (dm-0): recovery complete [ 5733.474958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.781395] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5739.003915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14721) [ 5739.003925] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14689) [ 5740.078850] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5740.581859] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5743.990952] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 19:26:07 (1775172367) [ 5746.757650] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5747.365489] Lustre: Failing over lustre-MDT0001 [ 5747.462626] Lustre: server umount lustre-MDT0001 complete [ 5761.356107] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5761.357749] LDISKFS-fs (dm-1): recovery complete [ 5761.361786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5762.717194] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5766.649901] Lustre: 208214:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed4102e2680 x1861397291341696/t42949673097(0) o36->1f1b116f-0937-4906-afca-85d7a7b8a129@192.168.203.41@tcp:545/0 lens 560/2880 e 0 to 0 dl 1775172400 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5766.651384] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4662 to 0x2c0000400:4705) [ 5766.651395] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4662 to 0x280000400:4705) [ 5766.658865] Lustre: 208214:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5767.714229] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5768.208104] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5771.554446] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 19:26:34 (1775172394) [ 5777.173692] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5779.449219] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5780.053900] Lustre: Failing over lustre-MDT0000 [ 5780.283614] Lustre: server umount lustre-MDT0000 complete [ 5794.047180] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5794.048812] LDISKFS-fs (dm-0): recovery complete [ 5794.051927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5795.333981] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5799.425840] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14721) [ 5799.425840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14753) [ 5800.547100] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5801.064978] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5802.928053] Lustre: Failing over lustre-MDT0001 [ 5803.016150] Lustre: server umount lustre-MDT0001 complete [ 5816.729172] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5816.730446] LDISKFS-fs (dm-1): recovery complete [ 5816.734119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5818.097718] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5821.939187] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4716 to 0x280000400:4737) [ 5821.939190] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4716 to 0x2c0000400:4737) [ 5822.982131] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5823.713136] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5827.050240] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 19:27:30 (1775172450) [ 5827.373150] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5827.376176] Lustre: Skipped 3 previous similar messages [ 5832.691811] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5835.113862] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5835.726039] Lustre: Failing over lustre-MDT0000 [ 5835.954544] Lustre: server umount lustre-MDT0000 complete [ 5837.127804] LustreError: 42370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775172460 with bad export cookie 453938260941291143 [ 5837.129168] Lustre: Failing over lustre-MDT0001 [ 5837.131133] LustreError: 42370:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5837.232588] Lustre: server umount lustre-MDT0001 complete [ 5851.712156] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5851.712536] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5851.713526] LDISKFS-fs (dm-0): recovery complete [ 5851.715509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.716600] LDISKFS-fs (dm-1): recovery complete [ 5851.732440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5853.664088] Lustre: 3642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775172460/real 1775172460] req@ffff8ed53c8eb800 x1861397238947968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775172476 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5853.670595] Lustre: 3642:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5861.856308] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ed53c8e8380 x1861397238950528/t0(0) o250->MGC192.168.203.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5863.291427] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5863.424473] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5864.069552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14785) [ 5864.069577] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14753) [ 5868.173587] Lustre: 218836:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed53d29ce00 x1861397291406720/t51539607620(0) o36->1f1b116f-0937-4906-afca-85d7a7b8a129@192.168.203.41@tcp:647/0 lens 560/2880 e 0 to 0 dl 1775172502 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5868.180783] Lustre: 218836:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5868.181422] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4748 to 0x280000400:4769) [ 5868.181460] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4748 to 0x2c0000400:4769) [ 5869.254827] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5869.768485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5870.237587] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5873.566678] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 19:28:16 (1775172496) [ 5873.907749] LustreError: 174384:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed53d22f800 x1861397238981120/t386547056658(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:653/0 lens 1744/4320 e 0 to 0 dl 1775172508 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5873.914715] LustreError: 174384:0:(ldlm_lib.c:3325:target_send_reply_msg()) Skipped 3 previous similar messages [ 5876.190169] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5876.771343] Lustre: Failing over lustre-MDT0001 [ 5882.927849] Lustre: server umount lustre-MDT0001 complete [ 5896.803818] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5896.805493] LDISKFS-fs (dm-1): recovery complete [ 5896.810710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5898.225706] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5902.322477] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4801) [ 5902.322540] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4801) [ 5903.373754] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5903.896463] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5907.160932] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 19:28:50 (1775172530) [ 5909.950328] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5910.549312] Lustre: Failing over lustre-MDT0000 [ 5910.795647] Lustre: server umount lustre-MDT0000 complete [ 5922.784270] Lustre: 221129:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775172530/real 1775172530] req@ffff8ed405c2aa00 x1861397239012352/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1488/4320 e 0 to 1 dl 1775172546 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5922.796349] Lustre: 221129:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 5924.792497] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5924.793848] LDISKFS-fs (dm-0): recovery complete [ 5924.797926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5926.268320] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5929.990992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14817) [ 5929.991080] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14785) [ 5931.226878] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5931.811363] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5935.394542] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 19:29:18 (1775172558) [ 5938.181052] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5940.674863] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5941.280519] Lustre: Failing over lustre-MDT0000 [ 5941.523249] Lustre: server umount lustre-MDT0000 complete [ 5955.251896] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5955.253712] LDISKFS-fs (dm-0): recovery complete [ 5955.255779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5956.507726] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5960.702212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14849) [ 5960.702232] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14817) [ 5961.727785] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5962.258092] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5964.152864] Lustre: Failing over lustre-MDT0001 [ 5964.232741] Lustre: server umount lustre-MDT0001 complete [ 5978.040879] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5978.042652] LDISKFS-fs (dm-1): recovery complete [ 5978.047208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.329075] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5983.219123] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4833) [ 5983.219127] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4833) [ 5984.298170] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5984.788054] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5987.984848] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 19:30:11 (1775172611) [ 5990.627947] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5993.017505] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5993.618184] Lustre: Failing over lustre-MDT0000 [ 5993.811183] Lustre: server umount lustre-MDT0000 complete [ 5994.943574] LustreError: 42369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775172618 with bad export cookie 453938260941301580 [ 5994.944929] Lustre: Failing over lustre-MDT0001 [ 5994.948700] LustreError: 42369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5998.560563] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.564289] Lustre: Skipped 67 previous similar messages [ 6001.120774] LustreError: 228318:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8ed53c3f5880 x1861397239092992/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6001.129893] LustreError: 228318:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6001.230618] Lustre: server umount lustre-MDT0001 complete [ 6015.595211] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6015.598261] LDISKFS-fs (dm-1): recovery complete [ 6015.602943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6015.603303] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6015.607458] LDISKFS-fs (dm-0): recovery complete [ 6015.611918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6015.678524] LustreError: 229415:0:(llog.c:1646:llog_backup()) MGC192.168.203.141@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6015.682024] Lustre: 229415:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.141@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6020.064303] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ed53c3f7100 x1861397239094144/t0(0) o250->MGC192.168.203.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6020.127523] LustreError: 174386:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6020.136297] LustreError: 174386:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 206 previous similar messages [ 6020.153562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6020.156618] Lustre: Skipped 21 previous similar messages [ 6020.164388] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6020.166288] Lustre: Skipped 21 previous similar messages [ 6020.237063] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 6020.239126] LustreError: Skipped 17 previous similar messages [ 6021.628585] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6021.757356] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6025.377970] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6025.378213] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6025.380277] Lustre: Skipped 21 previous similar messages [ 6025.383318] Lustre: Skipped 72 previous similar messages [ 6026.431696] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6026.434514] Lustre: Skipped 21 previous similar messages [ 6026.452992] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14849) [ 6026.453051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14881) [ 6030.545333] Lustre: 229469:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8ed511b50700 x1861397291481856/t64424509442(0) o36->1f1b116f-0937-4906-afca-85d7a7b8a129@192.168.203.41@tcp:79/0 lens 496/2888 e 0 to 0 dl 1775172689 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6030.551387] Lustre: 229469:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6030.551793] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4865) [ 6030.551838] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4865) [ 6031.628781] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6032.138091] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6032.606078] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6035.856161] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 19:30:58 (1775172658) [ 6038.795633] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6039.436136] Lustre: Failing over lustre-MDT0000 [ 6039.634475] Lustre: server umount lustre-MDT0000 complete [ 6053.434672] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6053.436065] LDISKFS-fs (dm-0): recovery complete [ 6053.439196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6054.816070] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6059.012671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14913) [ 6059.012736] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14881) [ 6060.141562] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6060.624530] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6091.266130] Lustre: lustre-MDT0001: Client 1f1b116f-0937-4906-afca-85d7a7b8a129 (at 192.168.203.41@tcp) reconnecting [ 6091.268748] Lustre: Skipped 1 previous similar message [ 6093.319629] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 19:31:56 (1775172716) [ 6093.637884] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6093.640158] Lustre: Skipped 5 previous similar messages [ 6095.882424] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6096.483134] Lustre: Failing over lustre-MDT0001 [ 6096.572631] Lustre: server umount lustre-MDT0001 complete [ 6110.366475] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6110.368556] LDISKFS-fs (dm-1): recovery complete [ 6110.373106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.745027] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6115.831691] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4897) [ 6115.831741] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4897) [ 6116.912960] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6117.402421] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6120.642856] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 19:32:23 (1775172743) [ 6123.186284] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6125.465797] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6126.053161] Lustre: Failing over lustre-MDT0000 [ 6126.171612] Lustre: server umount lustre-MDT0000 complete [ 6139.775531] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6139.777527] LDISKFS-fs (dm-0): recovery complete [ 6139.781099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.816108] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6139.818733] LustreError: Skipped 11 previous similar messages [ 6141.011307] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6145.022528] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14913) [ 6145.022554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14945) [ 6146.103363] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6146.602341] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6148.487272] Lustre: Failing over lustre-MDT0001 [ 6148.567686] Lustre: server umount lustre-MDT0001 complete [ 6162.324798] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6162.326598] LDISKFS-fs (dm-1): recovery complete [ 6162.331182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.614326] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6167.541134] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4929) [ 6167.541152] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4929) [ 6168.596951] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6169.108962] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6172.305678] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 19:33:15 (1775172795) [ 6172.633566] LustreError: 229425:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ffff8ed531717b80 x1861397291538176/t77309411330(0) o36->1f1b116f-0937-4906-afca-85d7a7b8a129@192.168.203.41@tcp:196/0 lens 496/456 e 0 to 0 dl 1775172806 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6172.640797] LustreError: 229425:0:(ldlm_lib.c:3325:target_send_reply_msg()) Skipped 6 previous similar messages [ 6174.917193] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6177.241863] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6178.150331] Lustre: Failing over lustre-MDT0000 [ 6178.307035] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping) [ 6178.308971] Lustre: Skipped 17 previous similar messages [ 6178.373210] Lustre: server umount lustre-MDT0000 complete [ 6179.550600] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775172802 with bad export cookie 453938260941309133 [ 6179.551962] Lustre: Failing over lustre-MDT0001 [ 6179.555203] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6179.649696] Lustre: server umount lustre-MDT0001 complete [ 6194.192072] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6194.194239] LDISKFS-fs (dm-0): recovery complete [ 6194.195171] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6194.197808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6194.198816] LDISKFS-fs (dm-1): recovery complete [ 6194.206381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6199.264119] Lustre: 3642:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775172806/real 1775172806] req@ffff8ed411652300 x1861397239239808/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775172822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6199.270815] Lustre: 3642:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 6204.384286] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ed405f81c00 x1861397239242368/t0(0) o250->MGC192.168.203.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6205.805499] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6205.955091] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6213.587433] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4780 to 0x2c0000400:4961) [ 6213.587444] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4780 to 0x280000400:4961) [ 6217.698318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14595 to 0x280000401:14977) [ 6217.698318] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14450 to 0x2c0000401:14945) [ 6218.950875] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6219.539586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6220.031941] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6223.506911] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 19:34:06 (1775172846) [ 6224.074680] Lustre: 241347:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 1f1b116f-0937-4906-afca-85d7a7b8a129 at adminstrative request [ 6228.612778] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 19:34:11 (1775172851) [ 6230.262865] Lustre: Failing over lustre-MDT0000 [ 6230.402186] Lustre: server umount lustre-MDT0000 complete [ 6242.921298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6244.405993] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6248.497896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:15028 to 0x280000401:15073) [ 6248.497918] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14997 to 0x2c0000401:15041) [ 6249.941619] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6250.610443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6254.317076] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 19:34:37 (1775172877) [ 6258.380054] Lustre: Failing over lustre-OST0000 [ 6258.445490] Lustre: server umount lustre-OST0000 complete [ 6271.766750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6273.602416] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6276.367272] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6276.856715] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6280.316664] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 19:35:03 (1775172903) [ 6281.628473] Lustre: Failing over lustre-MDT0000 [ 6281.857739] Lustre: server umount lustre-MDT0000 complete [ 6284.437374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6285.817521] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6286.568522] Lustre: lustre-MDT0000: Denying connection for new client 7715e2c1-17d7-407e-874d-67b4c2b33d79 (at 192.168.203.41@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 6286.573589] Lustre: Skipped 1 previous similar message [ 6289.909982] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14997 to 0x2c0000401:15073) [ 6289.910034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:15028 to 0x280000401:15105) [ 6294.269863] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 19:35:17 (1775172917) [ 6297.209358] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6298.265261] Lustre: Failing over lustre-OST0000 [ 6298.312590] Lustre: server umount lustre-OST0000 complete [ 6313.009932] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6313.011559] LDISKFS-fs (dm-2): recovery complete [ 6313.015410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6314.945390] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6318.068979] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6318.646953] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6322.152294] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 19:35:45 (1775172945) [ 6324.915280] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6326.808066] Lustre: Failing over lustre-OST0000 [ 6326.848163] Lustre: server umount lustre-OST0000 complete [ 6340.714986] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6340.716735] LDISKFS-fs (dm-2): recovery complete [ 6340.720040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6342.536827] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6342.555248] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.41@tcp inode [0x2000301a1:0x5:0x0] object 0x280000401:15107 extent [0-4194303]: client csum bf47ba16, server csum e63358c5 [ 6345.389978] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6345.846222] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6349.732170] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 19:36:12 (1775172972) [ 6352.505387] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6354.924816] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6357.029851] Lustre: Failing over lustre-MDT0000 [ 6357.290135] Lustre: server umount lustre-MDT0000 complete [ 6358.491538] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775172981 with bad export cookie 453938260941335250 [ 6358.494464] Lustre: Failing over lustre-OST0000 [ 6358.495996] LustreError: 6477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6358.536761] Lustre: server umount lustre-OST0000 complete [ 6373.083108] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6373.084466] LDISKFS-fs (dm-0): recovery complete [ 6373.087322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6383.072536] LustreError: 3639:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ed40a6b3100 x1861397239392768/t0(0) o250->MGC192.168.203.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6384.405366] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6388.315325] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14997 to 0x2c0000401:15105) [ 6399.797636] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6399.800632] LDISKFS-fs (dm-2): recovery complete [ 6399.807534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6402.390539] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6409.123858] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 19:37:12 (1775173032) [ 6414.813284] Lustre: Failing over lustre-OST0000 [ 6414.891371] Lustre: server umount lustre-OST0000 complete [ 6416.528288] Lustre: Failing over lustre-MDT0000 [ 6416.781988] Lustre: server umount lustre-MDT0000 complete [ 6429.846711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.128223] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6435.322068] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14997 to 0x2c0000401:15137) [ 6438.795362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6440.444449] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6441.087838] Lustre: lustre-OST0000: Denying connection for new client 70c6072d-67c7-4ca5-8810-011b3ae5caba (at 192.168.203.41@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 6441.092416] Lustre: Skipped 1 previous similar message [ 6446.594041] Lustre: lustre-OST0000: Denying connection for new client 70c6072d-67c7-4ca5-8810-011b3ae5caba (at 192.168.203.41@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 6446.598272] Lustre: Skipped 1 previous similar message [ 6456.834081] Lustre: lustre-OST0000: Denying connection for new client 70c6072d-67c7-4ca5-8810-011b3ae5caba (at 192.168.203.41@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 6456.838247] Lustre: Skipped 1 previous similar message [ 6477.313864] Lustre: lustre-OST0000: Denying connection for new client 70c6072d-67c7-4ca5-8810-011b3ae5caba (at 192.168.203.41@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 6477.320906] Lustre: Skipped 3 previous similar messages [ 6509.500110] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6509.501855] Lustre: 254399:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 7715e2c1-17d7-407e-874d-67b4c2b33d79@ [ 6509.504940] Lustre: 254399:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6509.507103] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6509.512472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:15156 to 0x280000401:15177) [ 6514.340901] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [ 6518.624957] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 6521.402151] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 19:39:04 (1775173144) [ 6522.811811] Lustre: Failing over lustre-OST0001 [ 6522.865414] Lustre: server umount lustre-OST0001 complete [ 6536.804751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6538.674265] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6543.344194] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 19:39:26 (1775173166) [ 6544.649682] Lustre: Failing over lustre-OST0000 [ 6544.710170] Lustre: server umount lustre-OST0000 complete [ 6558.324032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6560.315392] LustreError: 257375:0:(ldlm_lib.c:2884:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6560.321272] LustreError: 257375:0:(ldlm_lib.c:2884:target_recovery_thread()) Skipped 78 previous similar messages [ 6560.507791] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6566.368173] Lustre: *** cfs_fail_loc=715, val=40*** [ 6575.619320] Lustre: lustre-OST0000: Client 70c6072d-67c7-4ca5-8810-011b3ae5caba (at 192.168.203.41@tcp) reconnected, waiting for 3 clients in recovery for 1:24 [ 6576.608181] Lustre: 3639:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775173183/real 1775173183] req@ffff8ed4095ba300 x1861397239508224/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775173199 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 6576.625712] Lustre: 3639:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [ 6576.631907] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:23 [ 6576.638971] Lustre: Skipped 1 previous similar message [ 6581.728263] Lustre: *** cfs_fail_loc=715, val=40*** [ 6581.731100] Lustre: Skipped 2 previous similar messages [ 6591.969144] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 6591.975672] Lustre: Skipped 1 previous similar message [ 6598.112229] Lustre: *** cfs_fail_loc=715, val=40*** [ 6598.115736] Lustre: Skipped 2 previous similar messages [ 6600.368096] LustreError: 257375:0:(ldlm_lib.c:2884:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6600.374104] LustreError: 257375:0:(ldlm_lib.c:2884:target_recovery_thread()) Skipped 77 previous similar messages [ 6602.086124] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6602.591601] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6606.075816] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 19:40:29 (1775173229) [ 6607.849799] Lustre: Failing over lustre-MDT0000 [ 6608.148960] Lustre: server umount lustre-MDT0000 complete [ 6609.376920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6609.387235] Lustre: Skipped 45 previous similar messages [ 6621.699819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6621.873802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6621.877938] Lustre: Skipped 18 previous similar messages [ 6621.897354] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6621.900821] Lustre: Skipped 18 previous similar messages [ 6623.393169] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6627.301089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6627.304826] Lustre: Skipped 53 previous similar messages [ 6627.312673] LustreError: 258814:0:(ldlm_lib.c:2884:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6628.832085] Lustre: *** cfs_fail_loc=715, val=80*** [ 6628.833550] Lustre: Skipped 2 previous similar messages [ 6633.440228] Lustre: *** cfs_fail_loc=715, val=80*** [ 6639.136365] Lustre: lustre-MDT0000: Client 70c6072d-67c7-4ca5-8810-011b3ae5caba (at 192.168.203.41@tcp) reconnected, waiting for 2 clients in recovery for 1:28 [ 6639.143455] Lustre: Skipped 1 previous similar message [ 6643.680877] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6645.216174] Lustre: *** cfs_fail_loc=715, val=80*** [ 6655.490597] Lustre: lustre-MDT0000: Client 70c6072d-67c7-4ca5-8810-011b3ae5caba (at 192.168.203.41@tcp) reconnected, waiting for 2 clients in recovery for 1:12 [ 6660.064788] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6661.600238] Lustre: *** cfs_fail_loc=715, val=80*** [ 6661.603619] Lustre: Skipped 1 previous similar message [ 6670.850460] Lustre: lustre-MDT0000: Client 70c6072d-67c7-4ca5-8810-011b3ae5caba (at 192.168.203.41@tcp) reconnected, waiting for 2 clients in recovery for 0:56 [ 6676.448679] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6687.234301] Lustre: lustre-MDT0000: Client 70c6072d-67c7-4ca5-8810-011b3ae5caba (at 192.168.203.41@tcp) reconnected, waiting for 2 clients in recovery for 0:40 [ 6691.809058] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6697.952162] Lustre: *** cfs_fail_loc=715, val=80*** [ 6697.953422] Lustre: Skipped 4 previous similar messages [ 6707.400057] LustreError: 258814:0:(ldlm_lib.c:2884:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6707.405048] Lustre: 258814:0:(ldlm_lib.c:2930:target_recovery_thread()) too long recovery - read logs [ 6707.407491] LustreError: dumping log to /tmp/lustre-log.1775173330.258814 [ 6707.448731] Lustre: lustre-MDT0000: Recovery over after 1:25, of 2 clients 2 recovered and 0 were evicted. [ 6707.451041] Lustre: Skipped 18 previous similar messages [ 6707.465203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:15150 to 0x2c0000401:15169) [ 6707.465227] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:15191 to 0x280000401:15209) [ 6708.616630] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6709.079735] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6712.269264] Lustre: DEBUG MARKER: == replay-single test complete, duration 6557 sec ======== 19:42:15 (1775173335) [ 6712.719652] Lustre: DEBUG MARKER: === replay-single: start cleanup 19:42:15 (1775173335) === [ 6715.076123] Lustre: DEBUG MARKER: === replay-single: finish cleanup 19:42:18 (1775173338) === [ 6752.228753] Lustre: server umount lustre-MDT0000 complete [ 6753.248875] LustreError: 8381:0:(ldlm_lib.c:1178:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6753.259221] LustreError: 8381:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 195 previous similar messages [ 6755.727257] LustreError: 7415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775173379 with bad export cookie 453938260941348711 [ 6755.729871] LustreError: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6755.735045] LustreError: 7415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6755.748950] LustreError: Skipped 6 previous similar messages [ 6755.919211] Lustre: server umount lustre-MDT0001 complete [ 6769.098776] Lustre: server umount lustre-OST0000 complete [ 6782.365700] Lustre: server umount lustre-OST0001 complete [ 6787.347751] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing unload_modules_local [ 6788.249732] Key type lgssc unregistered [ 6788.368301] LNet: 261977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6788.370375] LNetError: 261977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6788.378306] LNet: Removed LNI 192.168.203.141@tcp [ 6788.649092] Key type .llcrypt unregistered [ 6788.650101] Key type ._llcrypt unregistered