[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491527228 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003106] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008311] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.012039] LSM: Security Framework initializing [ 0.013085] Yama: becoming mindful. [ 0.014034] SELinux: Initializing. [ 0.015069] *** VALIDATE selinux *** [ 0.020753] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025169] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028097] *** VALIDATE tmpfs *** [ 0.029407] *** VALIDATE proc *** [ 0.031027] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034195] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036038] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040388] debug: unmapping init [mem 0xffffffffb3659000-0xffffffffb3660fff] [ 0.042173] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043789] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046015] ... generic registers: 4 [ 0.047015] ... value mask: 0000ffffffffffff [ 0.048017] ... max period: 00007fffffffffff [ 0.049014] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051266] rcu: Hierarchical SRCU implementation. [ 0.053315] smp: Bringing up secondary CPUs ... [ 0.054487] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.058251] smp: Brought up 1 node, 4 CPUs [ 0.060013] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136361] node 0 deferred pages initialised in 73ms [ 0.140007] devtmpfs: initialized [ 0.141261] x86/mm: Memory block size: 128MB [ 0.143836] gcov: version magic: 0x41383552 [ 0.148200] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.149071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150239] pinctrl core: initialized pinctrl subsystem [ 0.151186] [ 0.151760] ************************************************************* [ 0.152014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153014] ** ** [ 0.154011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155012] ** ** [ 0.156015] ** This means that this kernel is built to expose internal ** [ 0.157013] ** IOMMU data structures, which may compromise security on ** [ 0.158014] ** your system. ** [ 0.159012] ** ** [ 0.160012] ** If you see this message and you are not debugging the ** [ 0.161011] ** kernel, report this immediately to your vendor! ** [ 0.162015] ** ** [ 0.163012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164012] ************************************************************* [ 0.165689] NET: Registered protocol family 16 [ 0.167472] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175663] cpuidle: using governor menu [ 0.178032] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181611] PCI: Using configuration type 1 for base access [ 0.185203] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197191] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.199108] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.203067] cryptd: max_cpu_qlen set to 1000 [ 0.207172] ACPI: Added _OSI(Module Device) [ 0.209012] ACPI: Added _OSI(Processor Device) [ 0.211012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218427] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224712] ACPI: Interpreter enabled [ 0.226057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228014] ACPI: Using IOAPIC for interrupt routing [ 0.230098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.245198] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.247062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.250022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254109] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259000] acpiphp: Slot [2] registered [ 0.260039] acpiphp: Slot [5] registered [ 0.261000] acpiphp: Slot [6] registered [ 0.261000] acpiphp: Slot [7] registered [ 0.263097] acpiphp: Slot [8] registered [ 0.264084] acpiphp: Slot [9] registered [ 0.266100] acpiphp: Slot [10] registered [ 0.267104] acpiphp: Slot [3] registered [ 0.269111] acpiphp: Slot [4] registered [ 0.270106] acpiphp: Slot [11] registered [ 0.271087] acpiphp: Slot [12] registered [ 0.273090] acpiphp: Slot [13] registered [ 0.274087] acpiphp: Slot [14] registered [ 0.276102] acpiphp: Slot [15] registered [ 0.277098] acpiphp: Slot [16] registered [ 0.279113] acpiphp: Slot [17] registered [ 0.280082] acpiphp: Slot [18] registered [ 0.282087] acpiphp: Slot [19] registered [ 0.284130] acpiphp: Slot [20] registered [ 0.285133] acpiphp: Slot [21] registered [ 0.287123] acpiphp: Slot [22] registered [ 0.288093] acpiphp: Slot [23] registered [ 0.290110] acpiphp: Slot [24] registered [ 0.292077] acpiphp: Slot [25] registered [ 0.293208] acpiphp: Slot [26] registered [ 0.295091] acpiphp: Slot [27] registered [ 0.296086] acpiphp: Slot [28] registered [ 0.297097] acpiphp: Slot [29] registered [ 0.299112] acpiphp: Slot [30] registered [ 0.301091] acpiphp: Slot [31] registered [ 0.302055] PCI host bridge to bus 0000:00 [ 0.303019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.306027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.308024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.314024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.319185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.322046] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.326322] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.336772] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.341938] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.344017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.347018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.349016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.352539] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.355785] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.359044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362790] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.383016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.387014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392740] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.401017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.421033] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454528] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.468016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.495015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.507720] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.515015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.523015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.552015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.567461] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.579013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.589011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.614021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.626430] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.634011] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.643015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.665015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.681824] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.690014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.698013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.717027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.729012] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.731368] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.734422] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.736438] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.739215] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.743130] iommu: Default domain type: Passthrough [ 0.744318] SCSI subsystem initialized [ 0.746134] ACPI: bus type USB registered [ 0.747087] usbcore: registered new interface driver usbfs [ 0.749074] usbcore: registered new interface driver hub [ 0.751082] usbcore: registered new device driver usb [ 0.753190] pps_core: LinuxPPS API ver. 1 registered [ 0.755014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.759081] PTP clock support registered [ 0.761123] EDAC MC: Ver: 3.0.0 [ 0.763156] PCI: Using ACPI for IRQ routing [ 0.766249] NetLabel: Initializing [ 0.768027] NetLabel: domain hash size = 128 [ 0.770011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.772076] NetLabel: unlabeled traffic allowed by default [ 0.775138] vgaarb: loaded [ 0.777325] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.779014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.787337] clocksource: Switched to clocksource kvm-clock [ 0.900896] VFS: Disk quotas dquot_6.6.0 [ 0.902351] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.904455] *** VALIDATE ramfs *** [ 0.905468] *** VALIDATE hugetlbfs *** [ 0.906733] pnp: PnP ACPI init [ 0.909563] pnp: PnP ACPI: found 6 devices [ 0.936117] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.939126] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.941300] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.943640] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.946046] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.948122] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.950765] NET: Registered protocol family 2 [ 0.953151] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.957928] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.962264] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.967670] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.970788] TCP: Hash tables configured (established 65536 bind 65536) [ 0.973540] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.975891] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.978240] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.981252] NET: Registered protocol family 1 [ 0.983530] RPC: Registered named UNIX socket transport module. [ 0.986830] RPC: Registered udp transport module. [ 0.989406] RPC: Registered tcp transport module. [ 0.991660] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.994225] NET: Registered protocol family 44 [ 0.996652] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.999015] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.001750] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.004239] PCI: CLS 0 bytes, default 64 [ 1.005787] Unpacking initramfs... [ 2.415941] debug: unmapping init [mem 0xffff8a45fcc54000-0xffff8a45fffbffff] [ 2.419043] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.420887] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.422796] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.909631] Initialise system trusted keyrings [ 2.911861] Key type blacklist registered [ 2.913755] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.923854] zbud: loaded [ 2.927204] *** VALIDATE nfs *** [ 2.928589] *** VALIDATE nfs4 *** [ 2.930237] pstore: using deflate compression [ 2.933901] Platform Keyring initialized [ 3.032690] NET: Registered protocol family 38 [ 3.034744] Key type asymmetric registered [ 3.036493] Asymmetric key parser 'x509' registered [ 3.038933] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.042275] io scheduler mq-deadline registered [ 3.044069] io scheduler kyber registered [ 3.046315] io scheduler bfq registered [ 3.048914] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.052205] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.055184] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.057752] ACPI: Power Button [PWRF] [ 3.062868] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.069665] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.083229] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.091477] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.106228] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.136851] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.168313] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.172951] Non-volatile memory driver v1.3 [ 3.174280] Linux agpgart interface v0.103 [ 3.204307] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.207521] vda: detected capacity change from 0 to 74702848 [ 3.221095] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.224590] vdb: detected capacity change from 0 to 1073741824 [ 3.239067] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.242494] vdc: detected capacity change from 0 to 2621440000 [ 3.256124] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.258964] vdd: detected capacity change from 0 to 2621440000 [ 3.271311] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.274301] vde: detected capacity change from 0 to 4294967296 [ 3.287281] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.290430] vdf: detected capacity change from 0 to 4294967296 [ 3.297122] libphy: Fixed MDIO Bus: probed [ 3.302427] usbcore: registered new interface driver usbserial_generic [ 3.305366] usbserial: USB Serial support registered for generic [ 3.307299] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.312552] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.314105] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.316224] mousedev: PS/2 mouse device common for all mice [ 3.318679] rtc_cmos 00:05: RTC can wake from S4 [ 3.321586] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.324772] rtc_cmos 00:05: registered as rtc0 [ 3.328051] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.328302] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.333176] intel_pstate: CPU model not supported [ 3.333323] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.339356] hid: raw HID events driver (C) Jiri Kosina [ 3.341532] usbcore: registered new interface driver usbhid [ 3.343292] usbhid: USB HID core driver [ 3.344698] drop_monitor: Initializing network drop monitor service [ 3.347229] Initializing XFRM netlink socket [ 3.348842] NET: Registered protocol family 10 [ 3.351383] Segment Routing with IPv6 [ 3.353050] NET: Registered protocol family 17 [ 3.355308] mpls_gso: MPLS GSO support [ 3.361676] RAS: Correctable Errors collector initialized. [ 3.363882] AVX version of gcm_enc/dec engaged. [ 3.365666] AES CTR mode by8 optimization enabled [ 3.434041] sched_clock: Marking stable (3434008806, 0)->(4350428082, -916419276) [ 3.437727] registered taskstats version 1 [ 3.440135] Loading compiled-in X.509 certificates [ 3.442148] zswap: loaded using pool lzo/zbud [ 3.467972] Key type big_key registered [ 3.480700] Key type encrypted registered [ 3.482887] ima: No TPM chip found, activating TPM-bypass! [ 3.484910] ima: Allocated hash algorithm: sha1 [ 3.486753] ima: No architecture policies found [ 3.488448] evm: Initialising EVM extended attributes: [ 3.490406] evm: security.selinux [ 3.491695] evm: security.ima [ 3.492728] evm: security.capability [ 3.494118] evm: HMAC attrs: 0x1 [ 3.496545] rtc_cmos 00:05: setting system clock to 2026-08-15 21:02:16 UTC (1786827736) [ 3.502716] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 3.506399] debug: unmapping init [mem 0xffffffffb3382000-0xffffffffb3658fff] [ 3.515236] Write protecting the kernel read-only data: 28672k [ 3.518599] debug: unmapping init [mem 0xffffffffb1a03000-0xffffffffb1bfffff] [ 3.521067] debug: unmapping init [mem 0xffffffffb2314000-0xffffffffb23fffff] [ 3.553410] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.563425] systemd[1]: Detected virtualization kvm. [ 3.565409] systemd[1]: Detected architecture x86-64. [ 3.567014] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.594653] systemd[1]: No hostname configured. [ 3.596588] systemd[1]: Set hostname to . [ 3.598647] random: systemd: uninitialized urandom read (16 bytes read) [ 3.601336] systemd[1]: Initializing machine ID from random generator. [ 3.728845] random: systemd: uninitialized urandom read (16 bytes read) [ 3.733231] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.738338] random: systemd: uninitialized urandom read (16 bytes read) [ 3.742196] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.747707] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.359444] device-mapper: uevent: version 1.0.3 [ 4.361526] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.609609] virtio_net virtio0 ens2: renamed from eth0 [ 5.613949] scsi host0: ata_piix [ 5.626267] scsi host1: ata_piix [ 5.638168] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.648734] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 5.678541] random: fast init done [ 10.473114] dracut-initqueue[589]: RTNETLINK answers: File exists [ 11.003527] random: crng init done [ 11.005646] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.219632] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.296434] printk: systemd: 25 output lines suppressed due to ratelimiting [ 14.987949] SELinux: Disabled at runtime. [ 15.115241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.132924] systemd[1]: Detected virtualization kvm. [ 15.143072] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.186419] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.192592] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.200195] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.207335] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.211869] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.225067] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.233459] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ 16.508355] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 17.342432] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.901337] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.990235] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.332622] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.348770] EDAC sbridge: Ver: 1.1.2 [ 21.149303] Key type dns_resolver registered [ 21.592643] NFS: Registering the id_resolver key type [ 21.595332] Key type id_resolver registered [ 21.599525] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg631-server login: [ 69.347150] libcfs: loading out-of-tree module taints kernel. [ 69.378292] Key type ._llcrypt registered [ 69.380320] Key type .llcrypt registered [ 69.456854] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_hostid [ 85.118537] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 86.408667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 86.420882] alg: No test for adler32 (adler32-zlib) [ 87.721350] Lustre: Lustre: Build Version: 2.17.57_3_g928f38d [ 88.456986] LNet: Added LNI 192.168.206.131@tcp [8/256/0/180] [ 90.184120] Key type lgssc registered [ 91.688994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.115528] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.140897] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing load_modules_local [ 155.970367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 156.016608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.394572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 157.425718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 157.523085] Lustre: lustre-MDT0000: new disk, initializing [ 157.581088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.592940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.272610] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.628851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.753183] Lustre: 6519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.783151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.786821] Lustre: Skipped 1 previous similar message [ 173.873761] Lustre: lustre-MDT0001: new disk, initializing [ 173.947852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.973976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.990443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.189343] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.649747] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.737528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.954695] Lustre: lustre-OST0000: new disk, initializing [ 191.960692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 191.970958] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.062494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.749749] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 199.700098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 199.706468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 199.760764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.561486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.659947] Lustre: lustre-OST0001: new disk, initializing [ 212.662959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.667532] Lustre: 9528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.717974] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.879399] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.235107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 220.249486] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 220.312673] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.476145] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 241.190512] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.897177] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing check_logdir /tmp/testlogs/ [ 252.984382] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing yml_node [ 258.535482] Lustre: DEBUG MARKER: Client: 2.17.57.3 [ 262.015613] Lustre: DEBUG MARKER: MDS: 2.17.57.3 [ 265.610324] Lustre: DEBUG MARKER: OSS: 2.17.57.3 [ 267.733818] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sat Aug 15 17:06:39 EDT 2026 [ 284.901244] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 286.675597] Lustre: DEBUG MARKER: === replay-single: start setup 17:06:59 (1786828019) === [ 291.492294] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing check_config_client /mnt/lustre [ 313.317973] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 316.378027] Lustre: 13338:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 320.265871] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 325.838662] Lustre: DEBUG MARKER: === replay-single: finish setup 17:07:37 (1786828057) === [ 327.864082] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 17:07:40 (1786828060) [ 334.498912] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 336.086779] Lustre: Failing over lustre-MDT0000 [ 336.303942] Lustre: server umount lustre-MDT0000 complete [ 337.888587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 337.897078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.405518] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.523329] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.534511] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 344.068316] LustreError: 9388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.650852] LustreError: 7525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.667454] LustreError: 7525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 353.778669] LustreError: 6531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 353.807201] LustreError: 6531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 354.784677] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828071/real 1786828071] req@ffff8a4543cac000 x1873624769345664/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786828087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 354.835990] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 356.754535] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 356.759612] LDISKFS-fs (dm-0): recovery complete [ 356.781823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 359.907037] LustreError: 6531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.921562] LustreError: 6531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 364.179239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 364.543572] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 368.583797] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 369.640729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 369.696508] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 377.157513] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 378.686053] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 386.429030] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 17:08:38 (1786828118) [ 388.233068] Lustre: Failing over lustre-OST0000 [ 388.351558] Lustre: server umount lustre-OST0000 complete [ 389.095988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 389.109889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.123952] Lustre: Skipped 2 previous similar messages [ 389.129816] LustreError: 8440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 389.143502] LustreError: 8440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 390.116016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 405.473908] LustreError: 8438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 405.493731] LustreError: 8438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 405.730592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 405.998149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 407.119433] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 407.209257] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 407.211299] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 407.219188] Lustre: Skipped 3 previous similar messages [ 411.515147] hrtimer: interrupt took 7050635 ns [ 412.085366] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 420.861448] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 423.022327] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 431.479976] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 17:09:24 (1786828164) [ 439.460495] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 441.916392] Lustre: Failing over lustre-MDT0000 [ 442.403826] Lustre: server umount lustre-MDT0000 complete [ 446.432865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 446.439639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.454906] Lustre: Skipped 1 previous similar message [ 446.462929] LustreError: 6531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.481916] LustreError: 6531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 462.241409] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828179/real 1786828179] req@ffff8a45434af800 x1873624769406208/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786828195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 462.296083] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 464.112815] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 464.117704] LDISKFS-fs (dm-0): recovery complete [ 464.125696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.812501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.178261] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 478.195495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 478.216508] Lustre: Skipped 1 previous similar message [ 478.544623] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 481.526108] Lustre: lustre-MDT0000: Denying connection for new client db969615-a426-49a9-834c-9b64f01fc652 (at 192.168.206.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 486.912237] Lustre: lustre-MDT0000: Denying connection for new client db969615-a426-49a9-834c-9b64f01fc652 (at 192.168.206.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [ 492.042048] Lustre: lustre-MDT0000: Denying connection for new client db969615-a426-49a9-834c-9b64f01fc652 (at 192.168.206.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:56 [ 497.148860] Lustre: lustre-MDT0000: Denying connection for new client db969615-a426-49a9-834c-9b64f01fc652 (at 192.168.206.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:51 [ 502.271290] Lustre: lustre-MDT0000: Denying connection for new client db969615-a426-49a9-834c-9b64f01fc652 (at 192.168.206.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:46 [ 512.514197] Lustre: lustre-MDT0000: Denying connection for new client db969615-a426-49a9-834c-9b64f01fc652 (at 192.168.206.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:35 [ 512.539758] Lustre: Skipped 1 previous similar message [ 532.991148] Lustre: lustre-MDT0000: Denying connection for new client db969615-a426-49a9-834c-9b64f01fc652 (at 192.168.206.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:15 [ 533.007351] Lustre: Skipped 3 previous similar messages [ 548.500305] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 548.508941] Lustre: 18240:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 74686c40-d418-4972-80ce-7727499dbd21@ [ 548.514252] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 548.526857] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 548.527085] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 548.535955] Lustre: Skipped 2 previous similar messages [ 548.554828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 548.555054] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 563.802558] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 17:11:35 (1786828295) [ 571.279250] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 573.699139] Lustre: Failing over lustre-MDT0000 [ 573.930639] Lustre: server umount lustre-MDT0000 complete [ 574.433542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 574.447080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 574.464630] Lustre: Skipped 3 previous similar messages [ 574.468446] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.494899] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 23 previous similar messages [ 591.843750] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828308/real 1786828308] req@ffff8a45434ac000 x1873624769467008/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786828324 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 591.863760] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 596.175911] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 596.178048] LDISKFS-fs (dm-0): recovery complete [ 596.183315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.062047] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e82855d57c [ 601.096382] Lustre: MGC192.168.206.131@tcp: Connection restored to 0@lo (at 0@lo) [ 601.380627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.424843] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 604.968228] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 606.697828] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 607.253645] Lustre: lustre-MDT0000: Denying connection for new client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 607.279234] Lustre: Skipped 3 previous similar messages [ 671.750371] Lustre: lustre-MDT0000: Denying connection for new client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:04 [ 671.769028] Lustre: Skipped 12 previous similar messages [ 676.503687] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 676.516585] Lustre: 19968:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client db969615-a426-49a9-834c-9b64f01fc652@ [ 676.532336] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 676.566114] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 676.566952] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 676.585173] Lustre: Skipped 3 previous similar messages [ 676.613802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 676.620048] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 685.364462] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 17:13:37 (1786828417) [ 692.191251] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 694.068838] Lustre: Failing over lustre-MDT0000 [ 694.277587] Lustre: server umount lustre-MDT0000 complete [ 697.314487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 697.321265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 697.338521] Lustre: Skipped 3 previous similar messages [ 703.975439] LustreError: 18215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.999759] LustreError: 18215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [ 713.869492] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 713.876094] LDISKFS-fs (dm-0): recovery complete [ 713.883141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.989809] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 714.151193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 714.188338] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 717.956369] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 718.849666] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 719.335883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 719.433433] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 719.481801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 719.482131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 726.730488] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 728.288440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 736.416317] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 17:14:28 (1786828468) [ 742.846976] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 744.414071] Lustre: Failing over lustre-MDT0000 [ 744.487454] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 744.565472] Lustre: server umount lustre-MDT0000 complete [ 744.928575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 744.931546] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 744.941700] Lustre: Skipped 5 previous similar messages [ 765.673347] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 765.676895] LDISKFS-fs (dm-0): recovery complete [ 765.687050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 765.811635] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 766.027106] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 767.532669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 769.886803] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 771.058538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 771.068615] Lustre: Skipped 3 previous similar messages [ 771.127462] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 771.167476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 771.168267] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 778.317966] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 779.722167] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 788.436592] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 17:15:20 (1786828520) [ 795.708992] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 797.682535] Lustre: Failing over lustre-MDT0000 [ 797.974857] Lustre: server umount lustre-MDT0000 complete [ 801.763907] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 801.768837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 801.772830] Lustre: Skipped 4 previous similar messages [ 818.147578] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828534/real 1786828534] req@ffff8a467759b800 x1873624769584512/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786828550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 818.175904] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 818.781655] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 818.783982] LDISKFS-fs (dm-0): recovery complete [ 818.792551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.613842] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 829.655338] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 832.176067] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 834.101822] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 834.135749] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:193) [ 834.150333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 840.093415] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 841.624395] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 848.883876] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 17:16:21 (1786828581) [ 855.153621] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 856.867785] Lustre: Failing over lustre-MDT0000 [ 857.162637] Lustre: server umount lustre-MDT0000 complete [ 875.488406] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828592/real 1786828592] req@ffff8a46817f9880 x1873624769618048/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786828608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 875.534079] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 877.647302] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 877.649255] LDISKFS-fs (dm-0): recovery complete [ 877.656614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.730734] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a45435cd180 x1873624769627648/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 886.056287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 886.062017] Lustre: Skipped 2 previous similar messages [ 886.092495] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 886.787917] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 890.055900] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 891.365152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 891.370727] Lustre: Skipped 7 previous similar messages [ 891.448572] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 891.496518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 891.497825] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 899.561370] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 901.534357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 910.554805] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 17:17:22 (1786828642) [ 918.027485] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 919.980596] Lustre: Failing over lustre-MDT0000 [ 920.311384] Lustre: server umount lustre-MDT0000 complete [ 922.087169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 922.087759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.092748] LustreError: Skipped 1 previous similar message [ 922.112287] Lustre: Skipped 6 previous similar messages [ 938.465991] Lustre: 3652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828655/real 1786828655] req@ffff8a468188b800 x1873624769654784/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786828671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 938.486601] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 942.589644] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 942.592124] LDISKFS-fs (dm-0): recovery complete [ 942.599914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 949.104809] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 953.940296] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 954.490974] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 954.492082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 965.077337] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 966.556162] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 976.082731] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 17:18:28 (1786828708) [ 977.071080] Lustre: *** cfs_fail_loc=13b, val=315*** [ 977.073080] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 977.083192] LustreError: 6529:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a45434ad500 x1873624754700416/t38654705666(0) o35->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:421/0 lens 392/456 e 0 to 0 dl 1786828721 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 984.833647] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 986.486437] Lustre: Failing over lustre-MDT0000 [ 986.764670] Lustre: server umount lustre-MDT0000 complete [ 990.177683] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 990.188411] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 138 previous similar messages [ 1006.560898] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828723/real 1786828723] req@ffff8a468188ad80 x1873624769695872/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786828739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1006.601268] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1007.771593] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1007.773912] LDISKFS-fs (dm-0): recovery complete [ 1007.784711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.800482] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a4543568700 x1873624769705472/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1017.132970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1017.744451] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1017.760387] Lustre: Skipped 1 previous similar message [ 1020.986638] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1022.438462] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1022.442507] Lustre: Skipped 7 previous similar messages [ 1022.470459] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1022.475744] Lustre: 6529:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a46774ec700 x1873624754700416/t38654705666(0) o35->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:466/0 lens 392/456 e 0 to 0 dl 1786828766 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1022.480936] Lustre: Skipped 1 previous similar message [ 1022.558295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1022.558320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1029.847318] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1031.471440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1040.484909] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 17:19:32 (1786828772) [ 1046.925525] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1048.572452] Lustre: Failing over lustre-MDT0000 [ 1048.782895] Lustre: server umount lustre-MDT0000 complete [ 1053.152850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1053.153489] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.168260] LustreError: Skipped 1 previous similar message [ 1053.196371] Lustre: Skipped 7 previous similar messages [ 1069.387070] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1069.389168] LDISKFS-fs (dm-0): recovery complete [ 1069.394394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.752937] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1073.990435] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1075.296473] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1075.297075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1082.474761] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1084.135939] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1092.358812] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 17:20:24 (1786828824) [ 1098.824755] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1099.636851] Lustre: *** cfs_fail_loc=114, val=0*** [ 1102.033613] Lustre: Failing over lustre-MDT0000 [ 1102.230627] Lustre: server umount lustre-MDT0000 complete [ 1121.252558] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828838/real 1786828838] req@ffff8a46780edf80 x1873624769764224/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786828854 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1122.996869] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1122.998904] LDISKFS-fs (dm-0): recovery complete [ 1123.005682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.489763] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a4644468700 x1873624769771648/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1135.862104] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1137.221137] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1137.224605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1145.837889] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1147.958915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1157.240022] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 17:21:29 (1786828889) [ 1164.593433] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1165.622346] Lustre: *** cfs_fail_loc=128, val=0*** [ 1168.459305] Lustre: Failing over lustre-MDT0000 [ 1168.793338] Lustre: server umount lustre-MDT0000 complete [ 1188.320884] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786828905/real 1786828905] req@ffff8a4681b7ea00 x1873624769799168/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786828921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1188.354559] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1188.366775] LustreError: Skipped 2 previous similar messages [ 1190.067279] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1190.076946] LDISKFS-fs (dm-0): recovery complete [ 1190.089761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.848858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.859026] Lustre: Skipped 4 previous similar messages [ 1198.889638] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1198.895262] Lustre: Skipped 1 previous similar message [ 1200.127896] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1200.137098] Lustre: Skipped 2 previous similar messages [ 1202.559588] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1204.252599] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1204.263877] Lustre: Skipped 2 previous similar messages [ 1204.335859] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1204.335859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1211.054340] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1212.809720] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1221.298196] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 17:22:33 (1786828953) [ 1228.729407] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1230.742963] Lustre: Failing over lustre-MDT0000 [ 1230.847489] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 1231.122193] Lustre: server umount lustre-MDT0000 complete [ 1252.973852] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1252.980775] LDISKFS-fs (dm-0): recovery complete [ 1252.992831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.078617] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e828560970 [ 1265.875347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1265.876107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1265.909921] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1276.474413] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1278.104947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1286.555989] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 17:23:38 (1786829018) [ 1293.296507] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1295.195448] Lustre: Failing over lustre-MDT0000 [ 1295.376683] Lustre: server umount lustre-MDT0000 complete [ 1316.936798] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1316.939645] LDISKFS-fs (dm-0): recovery complete [ 1316.947972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1327.389076] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1328.618995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1328.635448] Lustre: Skipped 20 previous similar messages [ 1328.823112] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1328.823413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1336.615463] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1338.396984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1346.648479] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 17:24:38 (1786829078) [ 1353.395589] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1360.714775] Lustre: Failing over lustre-MDT0000 [ 1360.986843] Lustre: server umount lustre-MDT0000 complete [ 1364.453967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.480205] Lustre: Skipped 20 previous similar messages [ 1380.705319] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786829097/real 1786829097] req@ffff8a4543cace00 x1873624769907584/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786829113 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1380.758041] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 1381.933982] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1381.937949] LDISKFS-fs (dm-0): recovery complete [ 1381.949861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.049776] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a4644565f80 x1873624769915904/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1390.335171] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1390.348671] Lustre: Skipped 2 previous similar messages [ 1394.558647] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1398.735487] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1398.737686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1404.160214] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1406.033909] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1429.463990] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 17:26:01 (1786829161) [ 1436.778621] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1438.589673] Lustre: Failing over lustre-MDT0000 [ 1438.839762] Lustre: server umount lustre-MDT0000 complete [ 1439.712779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1439.726604] LustreError: Skipped 1 previous similar message [ 1457.058435] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1457.068061] LustreError: Skipped 3 previous similar messages [ 1460.723345] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1460.725611] LDISKFS-fs (dm-0): recovery complete [ 1460.731667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.364593] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a4678576d80 x1873624770015104/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1468.926361] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1468.935799] Lustre: Skipped 3 previous similar messages [ 1471.936778] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1473.123279] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1473.133268] Lustre: Skipped 3 previous similar messages [ 1473.171195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 1473.175427] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 1480.869649] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1482.479555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1492.481282] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 17:27:04 (1786829224) [ 1499.242720] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1500.954679] Lustre: Failing over lustre-MDT0000 [ 1501.162106] Lustre: server umount lustre-MDT0000 complete [ 1503.716038] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1503.742644] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 244 previous similar messages [ 1523.114079] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1523.116193] LDISKFS-fs (dm-0): recovery complete [ 1523.126940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.704925] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1535.035883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1535.036581] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1543.327360] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1545.004492] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1552.703991] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 17:28:05 (1786829285) [ 1559.094354] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1560.743290] Lustre: Failing over lustre-MDT0000 [ 1560.999280] Lustre: server umount lustre-MDT0000 complete [ 1584.086913] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1584.089487] LDISKFS-fs (dm-0): recovery complete [ 1584.111770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.264633] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a464440df80 x1873624770085376/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1596.317953] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1597.019195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1597.019749] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1605.172350] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1606.866837] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1616.013646] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 17:29:08 (1786829348) [ 1622.908825] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1624.740440] Lustre: Failing over lustre-MDT0000 [ 1624.950752] Lustre: server umount lustre-MDT0000 complete [ 1643.474115] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786829360/real 1786829360] req@ffff8a4678072300 x1873624770111616/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786829376 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1643.502680] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1645.850601] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1645.854606] LDISKFS-fs (dm-0): recovery complete [ 1645.861314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1653.728436] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a4678505500 x1873624770120320/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1654.007781] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1654.012567] Lustre: Skipped 3 previous similar messages [ 1657.530796] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1659.528397] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1659.529615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1666.524973] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1667.866749] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1676.466225] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 17:30:08 (1786829408) [ 1684.712536] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1687.099415] Lustre: Failing over lustre-MDT0000 [ 1687.491272] Lustre: server umount lustre-MDT0000 complete [ 1708.218123] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1708.220162] LDISKFS-fs (dm-0): recovery complete [ 1708.228136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.910392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.914162] Lustre: Skipped 7 previous similar messages [ 1720.644236] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1722.452911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1722.453899] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1729.339300] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1730.707536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1738.281596] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 17:31:10 (1786829470) [ 1745.292224] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1747.113866] Lustre: Failing over lustre-MDT0000 [ 1749.319832] Lustre: server umount lustre-MDT0000 complete [ 1770.290415] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1770.295279] LDISKFS-fs (dm-0): recovery complete [ 1770.303417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1779.687896] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e82856ca6e [ 1783.987269] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1785.469051] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1785.470681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1793.972325] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1795.581703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1804.437924] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 17:32:16 (1786829536) [ 1812.021846] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1813.702459] Lustre: Failing over lustre-MDT0000 [ 1813.970485] Lustre: server umount lustre-MDT0000 complete [ 1834.585926] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1834.588698] LDISKFS-fs (dm-0): recovery complete [ 1834.599295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.848719] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1847.782132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1847.785868] Lustre: Skipped 32 previous similar messages [ 1847.877832] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1847.877832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1854.664341] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1856.684371] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1866.478243] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 17:33:18 (1786829598) [ 1874.629651] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1876.476141] Lustre: Failing over lustre-MDT0000 [ 1876.693608] Lustre: server umount lustre-MDT0000 complete [ 1878.498145] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1878.507023] Lustre: Skipped 30 previous similar messages [ 1897.881118] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1897.885630] LDISKFS-fs (dm-0): recovery complete [ 1897.899737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.684763] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1909.824574] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1909.827146] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1917.763339] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1919.016878] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1927.382915] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 17:34:19 (1786829659) [ 1934.549652] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1936.602279] Lustre: Failing over lustre-MDT0000 [ 1936.778601] Lustre: server umount lustre-MDT0000 complete [ 1958.820704] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1958.830694] LDISKFS-fs (dm-0): recovery complete [ 1958.838578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.560996] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a46442ae680 x1873624770293760/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1972.278836] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 1972.279586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 1972.785597] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1982.827620] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1984.975968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1994.199261] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 17:35:26 (1786829726) [ 2001.651477] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2003.643734] Lustre: Failing over lustre-MDT0000 [ 2003.938872] Lustre: server umount lustre-MDT0000 complete [ 2008.038079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2008.044605] LustreError: Skipped 7 previous similar messages [ 2024.418367] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2024.428299] LustreError: Skipped 8 previous similar messages [ 2026.238859] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2026.244584] LDISKFS-fs (dm-0): recovery complete [ 2026.261375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.660030] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a45434aea00 x1873624770331776/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2035.202219] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2035.206808] Lustre: Skipped 8 previous similar messages [ 2039.264730] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2040.363378] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2040.371165] Lustre: Skipped 8 previous similar messages [ 2040.408993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 2040.409353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:897) [ 2048.221569] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2049.647536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2057.713504] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 17:36:29 (1786829789) [ 2065.656305] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2067.655956] Lustre: Failing over lustre-MDT0000 [ 2067.960448] Lustre: server umount lustre-MDT0000 complete [ 2088.888310] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2088.891308] LDISKFS-fs (dm-0): recovery complete [ 2088.899588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.478217] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2102.860875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2102.865020] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2109.891245] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2111.371768] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2118.451099] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 17:37:30 (1786829850) [ 2124.629537] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2126.379439] Lustre: Failing over lustre-MDT0000 [ 2126.585211] Lustre: server umount lustre-MDT0000 complete [ 2128.354464] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.370949] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 337 previous similar messages [ 2146.681361] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2146.684591] LDISKFS-fs (dm-0): recovery complete [ 2146.692231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2155.008779] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (not set up) [ 2158.911794] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2160.674755] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 2160.675475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 2166.949632] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2168.348452] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2174.699340] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 17:38:27 (1786829907) [ 2180.824714] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2182.474793] Lustre: Failing over lustre-MDT0000 [ 2182.704602] Lustre: server umount lustre-MDT0000 complete [ 2202.592114] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786829919/real 1786829919] req@ffff8a4646761500 x1873624770425472/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786829935 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2202.623040] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 2204.052284] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2204.054349] LDISKFS-fs (dm-0): recovery complete [ 2204.062407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.836071] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a467fe9a680 x1873624770434816/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2212.850446] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 2213.047357] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2213.054465] Lustre: Skipped 8 previous similar messages [ 2217.312848] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2218.571469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:993) [ 2218.571886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 2225.752993] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2227.427435] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2237.090870] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 17:39:29 (1786829969) [ 2245.527192] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2247.758674] Lustre: Failing over lustre-MDT0000 [ 2248.016369] Lustre: server umount lustre-MDT0000 complete [ 2269.482684] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2269.486103] LDISKFS-fs (dm-0): recovery complete [ 2269.504679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2275.851288] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e82856f6e4 [ 2280.239912] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2281.572742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 2281.573769] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 2289.124608] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2291.241987] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2299.146343] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 17:40:31 (1786830031) [ 2306.763859] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2308.729567] Lustre: Failing over lustre-MDT0000 [ 2309.012820] Lustre: server umount lustre-MDT0000 complete [ 2331.163632] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2331.165119] LDISKFS-fs (dm-0): recovery complete [ 2331.169714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2338.790918] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e82856fde4 [ 2339.030517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2339.038777] Lustre: Skipped 9 previous similar messages [ 2343.650217] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2344.541829] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 2344.542605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 2353.686326] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2355.371737] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2364.666510] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 17:41:37 (1786830097) [ 2371.524972] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2373.540631] Lustre: Failing over lustre-MDT0000 [ 2373.762871] Lustre: server umount lustre-MDT0000 complete [ 2395.296952] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2395.299976] LDISKFS-fs (dm-0): recovery complete [ 2395.310411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.980419] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2407.471266] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 2407.472390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 2414.842459] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2416.146734] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2423.449320] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 17:42:36 (1786830156) [ 2427.591477] Lustre: 75329:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 0c1246a1-a3e7-41e8-83f3-2e40bbd471be at adminstrative request [ 2431.555409] Lustre: Failing over lustre-MDT0000 [ 2431.785553] Lustre: server umount lustre-MDT0000 complete [ 2449.114320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.768507] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2463.735585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2463.749165] Lustre: Skipped 41 previous similar messages [ 2463.833555] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 2463.836243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 2471.885259] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2473.315902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2478.728125] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2487.139357] Lustre: DEBUG MARKER: before 3156, after 3156 [ 2492.391514] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 17:43:45 (1786830225) [ 2493.362264] Lustre: 77356:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 0c1246a1-a3e7-41e8-83f3-2e40bbd471be at adminstrative request [ 2502.752752] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 17:43:55 (1786830235) [ 2509.554793] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2511.362474] Lustre: Failing over lustre-MDT0000 [ 2511.587608] Lustre: server umount lustre-MDT0000 complete [ 2514.912968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.928079] Lustre: Skipped 39 previous similar messages [ 2532.400509] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2532.402720] LDISKFS-fs (dm-0): recovery complete [ 2532.409573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2540.519642] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e828570e22 [ 2545.641087] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2546.384569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 2546.385209] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 2554.614366] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2556.153283] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2565.687589] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 17:44:57 (1786830297) [ 2573.746680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2575.546849] Lustre: Failing over lustre-MDT0000 [ 2575.789842] Lustre: server umount lustre-MDT0000 complete [ 2595.448095] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2595.454747] LDISKFS-fs (dm-0): recovery complete [ 2595.473733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.497666] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a464446c700 x1873624770654464/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2603.532395] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 28 previous similar messages [ 2607.766991] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2609.215092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 2609.215117] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 2616.418446] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2617.912427] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2625.562500] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 17:45:57 (1786830357) [ 2632.826758] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2634.735391] Lustre: Failing over lustre-MDT0000 [ 2634.984139] Lustre: server umount lustre-MDT0000 complete [ 2655.202201] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2655.212300] LustreError: Skipped 9 previous similar messages [ 2656.560944] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2656.563102] LDISKFS-fs (dm-0): recovery complete [ 2656.579115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.009619] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2667.016993] Lustre: Skipped 9 previous similar messages [ 2670.164755] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2671.171857] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2671.186598] Lustre: Skipped 9 previous similar messages [ 2671.234065] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 2671.237031] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 2679.542340] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2681.282220] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2691.004527] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 17:47:03 (1786830423) [ 2698.902778] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2701.181905] Lustre: Failing over lustre-MDT0000 [ 2701.712250] Lustre: server umount lustre-MDT0000 complete [ 2724.053725] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2724.056370] LDISKFS-fs (dm-0): recovery complete [ 2724.065061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.417128] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.445500] LustreError: 6530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 325 previous similar messages [ 2733.576316] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (not set up) [ 2737.818794] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2739.256941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 2739.257467] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 2746.377460] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2748.013588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2756.875695] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 17:48:08 (1786830488) [ 2763.680550] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2765.351598] Lustre: Failing over lustre-MDT0000 [ 2765.520574] Lustre: server umount lustre-MDT0000 complete [ 2769.894403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2769.903339] LustreError: Skipped 7 previous similar messages [ 2790.998524] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2791.001264] LDISKFS-fs (dm-0): recovery complete [ 2791.012121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2800.434781] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2801.207558] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 2801.214078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 2809.641709] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2811.387812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2820.473970] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 17:49:12 (1786830552) [ 2827.689347] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2830.035298] Lustre: Failing over lustre-MDT0000 [ 2830.284104] Lustre: server umount lustre-MDT0000 complete [ 2847.696123] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786830564/real 1786830564] req@ffff8a4646446680 x1873624770785408/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786830580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2847.734844] Lustre: 3654:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 2851.870243] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2851.873334] LDISKFS-fs (dm-0): recovery complete [ 2851.884980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.962126] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e828572996 [ 2858.147970] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2858.152517] Lustre: Skipped 9 previous similar messages [ 2861.052758] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2863.683474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2863.684862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2868.923286] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2870.186440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2876.867380] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 17:50:09 (1786830609) [ 2883.521648] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2885.370138] Lustre: Failing over lustre-MDT0000 [ 2885.563592] Lustre: server umount lustre-MDT0000 complete [ 2906.387922] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2906.393235] LDISKFS-fs (dm-0): recovery complete [ 2906.404254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.818810] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e828572f85 [ 2919.111805] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2921.538638] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 2921.541844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 2926.643594] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2928.090503] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2935.068300] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 17:51:07 (1786830667) [ 2941.337685] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2943.095584] Lustre: Failing over lustre-MDT0000 [ 2943.350467] Lustre: server umount lustre-MDT0000 complete [ 2963.348138] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2963.351252] LDISKFS-fs (dm-0): recovery complete [ 2963.358375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.855342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.858398] Lustre: Skipped 9 previous similar messages [ 2977.550947] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2979.409430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 2979.412747] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 2985.648279] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2986.873733] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2994.956138] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 17:52:07 (1786830727) [ 3001.194676] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3003.147677] Lustre: Failing over lustre-MDT0000 [ 3003.371689] Lustre: server umount lustre-MDT0000 complete [ 3025.456493] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3025.460796] LDISKFS-fs (dm-0): recovery complete [ 3025.467798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.892671] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3036.202361] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3036.203424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3043.223906] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3044.643663] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3052.309166] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 17:53:04 (1786830784) [ 3058.867715] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3060.495399] Lustre: Failing over lustre-MDT0000 [ 3060.719639] Lustre: server umount lustre-MDT0000 complete [ 3081.785497] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3081.788032] LDISKFS-fs (dm-0): recovery complete [ 3081.801686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3091.757891] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3092.966776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3092.974813] Lustre: Skipped 42 previous similar messages [ 3093.044103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 3093.048300] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 3100.207489] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3102.052054] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3111.571780] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 17:54:03 (1786830843) [ 3118.758752] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3120.356453] Lustre: Failing over lustre-MDT0000 [ 3120.783392] Lustre: server umount lustre-MDT0000 complete [ 3123.681573] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.706859] Lustre: Skipped 40 previous similar messages [ 3142.099961] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3142.102730] LDISKFS-fs (dm-0): recovery complete [ 3142.141948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.311516] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3156.045436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 3156.046597] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 3163.678406] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3165.571398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3174.785490] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 17:55:06 (1786830906) [ 3176.010242] Lustre: 98930:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 0c1246a1-a3e7-41e8-83f3-2e40bbd471be at adminstrative request [ 3187.852599] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 17:55:19 (1786830919) [ 3195.446591] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3196.932727] Lustre: Failing over lustre-MDT0000 [ 3197.120807] Lustre: server umount lustre-MDT0000 complete [ 3209.102825] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3209.109833] LDISKFS-fs (dm-0): recovery complete [ 3209.120877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.455590] Lustre: lustre-MDT0000: Aborting client recovery [ 3209.459522] LustreError: 100212:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3209.464599] Lustre: 100245:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3209.470510] Lustre: 100245:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be@ [ 3209.479721] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3209.485568] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 3209.497097] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 3209.535079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 3209.543202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 3214.102516] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3214.818459] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3234.827742] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 17:56:07 (1786830967) [ 3241.311601] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3243.059307] Lustre: Failing over lustre-MDT0000 [ 3243.310415] Lustre: server umount lustre-MDT0000 complete [ 3256.030888] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3256.033869] LDISKFS-fs (dm-0): recovery complete [ 3256.042881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.131833] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3256.140311] LustreError: Skipped 9 previous similar messages [ 3256.396639] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3256.420978] Lustre: lustre-MDT0000: Aborting client recovery [ 3256.426981] LustreError: 102077:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3256.431879] Lustre: 102111:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3256.439896] Lustre: 102111:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3256.446298] Lustre: 102111:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3256.454767] Lustre: 102111:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3256.464041] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3256.470202] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 3256.480761] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 3256.538120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 3256.539474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 3261.051441] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3261.416690] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3274.171896] Lustre: *** cfs_fail_loc=1311, val=0*** [ 3274.176101] Lustre: Skipped 1 previous similar message [ 3280.684624] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 17:56:53 (1786831013) [ 3287.380727] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3289.404056] Lustre: Failing over lustre-MDT0000 [ 3289.765151] Lustre: server umount lustre-MDT0000 complete [ 3303.558555] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3303.566177] LDISKFS-fs (dm-0): recovery complete [ 3303.573641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3303.971153] Lustre: lustre-MDT0000: Aborting client recovery [ 3303.974535] LustreError: 103947:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3303.981091] Lustre: 103980:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3303.988919] Lustre: 103980:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3303.996345] Lustre: 103980:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3304.003616] Lustre: 103980:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3304.009280] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3304.016629] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 3304.040976] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 3304.090347] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 3304.097673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 3308.736863] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3309.039227] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3330.132819] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 17:57:42 (1786831062) [ 3331.080404] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3331.085600] LustreError: 14342:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a4681889500 x1873624755782272/t201863462916(0) o36->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:510/0 lens 512/456 e 0 to 0 dl 1786831075 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 3334.777574] Lustre: Failing over lustre-MDT0000 [ 3335.107799] Lustre: server umount lustre-MDT0000 complete [ 3336.728317] LustreError: 18215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3336.744265] LustreError: 18215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 266 previous similar messages [ 3343.284418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.600986] Lustre: lustre-MDT0000: Aborting client recovery [ 3343.603419] LustreError: 105272:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3343.609591] Lustre: 105305:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3343.616558] Lustre: 105305:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3343.621642] Lustre: 105305:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3343.634916] Lustre: 105305:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3343.639388] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3343.648858] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 3343.665549] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 3343.716885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 3343.718186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1601) [ 3348.575267] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3348.967474] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3369.706121] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 3371.509820] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 17:58:23 (1786831103) [ 3378.637852] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3380.692051] Lustre: Failing over lustre-MDT0000 [ 3380.854760] Lustre: server umount lustre-MDT0000 complete [ 3384.814063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3384.822905] LustreError: Skipped 7 previous similar messages [ 3390.568912] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3390.571336] LDISKFS-fs (dm-0): recovery complete [ 3390.580865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.848349] Lustre: lustre-MDT0000: Aborting client recovery [ 3390.852111] LustreError: 107236:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3390.857131] Lustre: 107268:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3390.864878] Lustre: 107268:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 3390.870215] Lustre: 107268:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3390.877852] Lustre: 107268:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3390.881740] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3390.885885] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 3390.895337] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 3390.940327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1633) [ 3390.949388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 3394.320784] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3396.067889] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3412.678167] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 17:59:05 (1786831145) [ 3444.457929] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3446.143859] Lustre: Failing over lustre-MDT0000 [ 3446.435903] Lustre: server umount lustre-MDT0000 complete [ 3463.649156] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786831180/real 1786831180] req@ffff8a454ab2ce00 x1873624771216512/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786831196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3463.678499] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 3467.736191] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3467.738521] LDISKFS-fs (dm-0): recovery complete [ 3467.746267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3473.889074] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a4677450700 x1873624771224960/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3473.933237] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [ 3474.256192] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3474.261886] Lustre: Skipped 20 previous similar messages [ 3475.968284] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3475.978769] Lustre: Skipped 8 previous similar messages [ 3478.425896] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3479.630530] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3479.638721] Lustre: Skipped 8 previous similar messages [ 3479.674139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 3479.675143] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 3486.083388] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3487.137174] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3505.448897] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 18:00:38 (1786831238) [ 3527.847472] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3536.707698] Lustre: Failing over lustre-MDT0000 [ 3536.960939] Lustre: server umount lustre-MDT0000 complete [ 3556.277879] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3556.279768] LDISKFS-fs (dm-0): recovery complete [ 3556.288273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.406224] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3574.614814] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 3574.614920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 3579.292437] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3581.008742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3603.377426] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 18:02:15 (1786831335) [ 3605.321301] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3606.227559] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3606.232357] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3612.260476] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 18:02:24 (1786831344) [ 3635.322739] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3643.819882] Lustre: Failing over lustre-OST0000 [ 3643.885748] Lustre: server umount lustre-OST0000 complete [ 3664.507835] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3664.514855] LDISKFS-fs (dm-2): recovery complete [ 3664.524839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3664.700514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3664.709279] Lustre: Skipped 10 previous similar messages [ 3669.705208] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3727.047646] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 18:04:19 (1786831459) [ 3733.055944] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3735.495205] Lustre: Failing over lustre-MDT0000 [ 3735.824245] Lustre: server umount lustre-MDT0000 complete [ 3738.595554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.619418] Lustre: Skipped 31 previous similar messages [ 3756.380516] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3756.383234] LDISKFS-fs (dm-0): recovery complete [ 3756.388263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.517587] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3771.881169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3771.886499] Lustre: Skipped 38 previous similar messages [ 3771.953451] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 3771.953664] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 3776.880220] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3778.206629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3787.236249] LustreError: 116689:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 3787.239495] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3788.256598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 3796.132581] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 18:05:28 (1786831528) [ 3801.110141] LustreError: 6526:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3806.176251] LustreError: 6526:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3806.186076] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnecting [ 3806.250327] LustreError: 43897:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 waking [ 3807.948044] LustreError: 9388:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3813.345362] LustreError: 9388:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3813.356521] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnecting [ 3814.869421] LustreError: 6527:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3820.002906] LustreError: 6527:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3820.009105] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnecting [ 3821.657098] LustreError: 9388:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3827.169417] LustreError: 9388:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3828.698771] LustreError: 14342:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3833.824228] LustreError: 14342:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3833.831079] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnecting [ 3833.840362] Lustre: Skipped 1 previous similar message [ 3841.792146] LustreError: 6526:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3841.797616] LustreError: 6526:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3847.136178] LustreError: 6526:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3847.141790] LustreError: 6526:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 3853.792469] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnecting [ 3853.801889] Lustre: Skipped 2 previous similar messages [ 3862.194607] LustreError: 6526:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 3862.201469] LustreError: 6526:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3867.616095] LustreError: 6526:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 3867.629292] LustreError: 6526:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 3875.486296] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 18:06:48 (1786831608) [ 3877.153272] LustreError: 14342:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3886.594088] Lustre: lustre-MDT0000: Export ffff8a46460ab000 already connecting from 192.168.206.31@tcp [ 3891.709895] Lustre: lustre-MDT0000: Export ffff8a46460ab000 already connecting from 192.168.206.31@tcp [ 3896.832560] Lustre: lustre-MDT0000: Export ffff8a46460ab000 already connecting from 192.168.206.31@tcp [ 3899.239354] Lustre: lustre-MDT0000: Export ffff8a46460ab000 already connecting from 192.168.206.31@tcp [ 3907.068369] Lustre: lustre-MDT0000: Export ffff8a46460ab000 already connecting from 192.168.206.31@tcp [ 3907.076722] Lustre: Skipped 1 previous similar message [ 3917.168288] LustreError: 14342:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3917.178568] Lustre: 14342:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8a4646588380 x1873624758441600/t0(0) o38->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1786831630 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3917.308776] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnecting [ 3917.315224] Lustre: Skipped 3 previous similar messages [ 3917.318297] LustreError: 6526:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3942.912831] Lustre: lustre-MDT0000: Export ffff8a46460ab000 already connecting from 192.168.206.31@tcp [ 3942.924987] Lustre: Skipped 1 previous similar message [ 3957.360213] LustreError: 6526:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3957.368417] Lustre: 6526:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8a467876f480 x1873624758445568/t0(0) o38->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1786831670 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3958.271813] LustreError: 18215:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3983.871925] Lustre: lustre-MDT0000: Export ffff8a46460ab000 already connecting from 192.168.206.31@tcp [ 3983.887950] Lustre: Skipped 3 previous similar messages [ 3998.361891] LustreError: 18215:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3998.370417] Lustre: 18215:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8a45434ac700 x1873624758448768/t0(0) o38->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1786831711 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3999.235323] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnecting [ 3999.241196] Lustre: Skipped 1 previous similar message [ 3999.243388] LustreError: 9388:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4023.809686] Lustre: lustre-MDT0000: Export ffff8a46460ab000 already connecting from 192.168.206.31@tcp [ 4023.820654] Lustre: Skipped 3 previous similar messages [ 4039.328850] LustreError: 9388:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4039.357021] Lustre: 9388:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8a45435cca80 x1873624758451968/t0(0) o38->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1786831752 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4044.295161] LustreError: 6527:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4084.336328] LustreError: 6527:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 awake [ 4084.346133] Lustre: 6527:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8a46442ca300 x1873624758455552/t0(0) o38->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:0/0 lens 520/416 e 0 to 0 dl 1786831797 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4085.248629] LustreError: 6526:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4099.843107] LustreError: 6526:0:(ldlm_lib.c:1436:target_handle_connect()) cfs_fail_timeout interrupted [ 4104.438835] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 18:10:36 (1786831836) [ 4111.271359] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4115.158611] Lustre: Failing over lustre-MDT0000 [ 4115.486286] Lustre: server umount lustre-MDT0000 complete [ 4116.007892] LustreError: 18215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4116.037301] LustreError: 18215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 128 previous similar messages [ 4120.043915] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4120.048311] LustreError: Skipped 3 previous similar messages [ 4127.182282] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4127.184836] LDISKFS-fs (dm-0): recovery complete [ 4127.191388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.327974] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4127.339393] LustreError: Skipped 6 previous similar messages [ 4127.556380] Lustre: *** cfs_fail_loc=712, val=0*** [ 4127.558742] LustreError: 43899:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8a4543568e00 x1873624771922944/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 4127.647390] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4127.647665] Lustre: lustre-MDT0000: Aborting client recovery [ 4127.651806] Lustre: Skipped 3 previous similar messages [ 4127.655897] LustreError: 121541:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4127.660524] Lustre: 121574:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4127.665719] Lustre: 121574:0:(ldlm_lib.c:2404:target_recovery_overseer()) Skipped 2 previous similar messages [ 4127.670505] Lustre: 121574:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4127.676192] Lustre: 121574:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4127.682227] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4127.688469] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 4127.698922] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 4127.738824] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 4127.741050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 4131.237661] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4132.844788] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4146.678078] Lustre: Failing over lustre-MDT0000 [ 4146.707843] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 4146.858966] Lustre: server umount lustre-MDT0000 complete [ 4164.578596] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786831881/real 1786831881] req@ffff8a4679f0c380 x1873624771939584/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786831897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4164.611751] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4165.440610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.726755] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4176.737301] Lustre: Skipped 3 previous similar messages [ 4179.430744] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4180.467837] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4180.479038] Lustre: Skipped 3 previous similar messages [ 4180.526815] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 4180.528148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 4188.118968] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4189.689682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4197.317314] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 18:12:09 (1786831929) [ 4197.530984] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnecting [ 4197.540357] Lustre: Skipped 2 previous similar messages [ 4203.649854] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 18:12:16 (1786831936) [ 4204.363919] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 4204.369725] LustreError: 6534:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a46785b0e00 x1873624758517888/t0(0) o700->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:628/0 lens 264/248 e 0 to 0 dl 1786831948 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 4223.675572] Lustre: Failing over lustre-MDT0000 [ 4223.915416] Lustre: server umount lustre-MDT0000 complete [ 4241.616305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4245.723994] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4247.084315] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 4247.084341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 4253.699740] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4255.069090] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4264.362195] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:13:16 (1786831996) [ 4266.638103] Lustre: Failing over lustre-OST0000 [ 4266.867281] Lustre: server umount lustre-OST0000 complete [ 4284.391411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4284.533491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4284.538396] Lustre: Skipped 4 previous similar messages [ 4290.216576] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4298.480509] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4299.843768] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4371.037171] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 18:15:03 (1786832103) [ 4378.478590] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4380.363246] Lustre: Failing over lustre-MDT0000 [ 4380.595493] Lustre: server umount lustre-MDT0000 complete [ 4382.176975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4382.197376] Lustre: Skipped 21 previous similar messages [ 4401.842742] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4401.848884] LDISKFS-fs (dm-0): recovery complete [ 4401.855236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4408.804031] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a46467df480 x1873624772077696/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4412.456308] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4414.448808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4414.460297] Lustre: Skipped 18 previous similar messages [ 4414.792071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 4414.797136] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 4484.051276] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 18:16:56 (1786832216) [ 4485.917098] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4485.922143] Lustre: Skipped 2 previous similar messages [ 4497.689582] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 18:17:10 (1786832230) [ 4501.018915] Lustre: Failing over lustre-MDT0000 [ 4501.486254] Lustre: server umount lustre-MDT0000 complete [ 4518.408188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4521.434505] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4524.022894] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 4524.028130] LustreError: 129297:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a45423bdf80 x1873624758646784/t0(0) o101->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:231/0 lens 328/344 e 0 to 0 dl 1786832306 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 4578.812600] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [ 4578.832513] Lustre: 129297:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4578.904344] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 4578.908675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 4582.833965] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4583.948327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4591.082948] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 18:18:43 (1786832323) [ 4593.026192] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4599.732525] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4601.239276] Lustre: Failing over lustre-MDT0000 [ 4601.405718] Lustre: server umount lustre-MDT0000 complete [ 4621.109645] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4621.112754] LDISKFS-fs (dm-0): recovery complete [ 4621.122302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.282416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 4621.291875] Lustre: Skipped 3 previous similar messages [ 4624.937441] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4626.459200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 4626.459459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 4633.190556] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4634.427541] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4640.438529] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 18:19:33 (1786832373) [ 4641.149315] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4647.636682] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4649.186844] Lustre: Failing over lustre-MDT0000 [ 4649.352672] Lustre: server umount lustre-MDT0000 complete [ 4668.117784] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4668.120137] LDISKFS-fs (dm-0): recovery complete [ 4668.125707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4671.815545] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4673.568544] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3137) [ 4673.570823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 4679.337783] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4680.680330] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4689.986395] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 18:20:22 (1786832422) [ 4691.548914] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4693.747579] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4701.823720] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4703.904524] Lustre: Failing over lustre-MDT0000 [ 4704.241256] Lustre: server umount lustre-MDT0000 complete [ 4719.102943] LustreError: 14342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4719.113237] LustreError: 14342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 176 previous similar messages [ 4724.302628] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4724.304656] LDISKFS-fs (dm-0): recovery complete [ 4724.311112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4730.852403] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e8285bdff0 [ 4731.067537] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4731.079221] Lustre: Skipped 9 previous similar messages [ 4734.752823] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4736.564781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3201) [ 4736.564781] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 4743.843896] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 18:21:16 (1786832476) [ 4745.910892] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4745.917480] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4745.922759] LustreError: 6529:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a4646799180 x1873624758704512/t257698037777(0) o35->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:414/0 lens 392/456 e 0 to 0 dl 1786832489 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4748.561902] Lustre: Failing over lustre-MDT0000 [ 4748.862041] Lustre: server umount lustre-MDT0000 complete [ 4751.846058] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4751.856041] LustreError: Skipped 6 previous similar messages [ 4765.654762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.770705] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4765.777556] LustreError: Skipped 7 previous similar messages [ 4769.889656] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4771.369356] Lustre: 6528:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a46787ee680 x1873624758704512/t257698037777(0) o35->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:440/0 lens 392/456 e 0 to 0 dl 1786832515 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4771.388587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3233) [ 4771.392932] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 4778.237945] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4779.626788] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4786.683743] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 18:21:59 (1786832519) [ 4787.543623] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4787.548460] LustreError: 18215:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a45423bc000 x1873624758719104/t261993005072(0) o36->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:500/0 lens 504/448 e 0 to 0 dl 1786832575 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4794.689594] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4796.259967] Lustre: Failing over lustre-MDT0000 [ 4796.373072] LustreError: 138102:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8a4655860e00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 4796.445861] Lustre: server umount lustre-MDT0000 complete [ 4816.568742] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4816.570453] LDISKFS-fs (dm-0): recovery complete [ 4816.576337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4818.430021] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4818.444184] Lustre: Skipped 8 previous similar messages [ 4820.735186] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4822.017744] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4822.028729] Lustre: Skipped 8 previous similar messages [ 4822.066142] Lustre: 18215:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a4644608380 x1873624758719104/t261993005072(0) o36->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:535/0 lens 504/2880 e 0 to 0 dl 1786832610 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4822.070792] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 4822.071102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3265) [ 4828.488751] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4829.869636] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4837.819698] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 18:22:49 (1786832569) [ 4838.962308] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4838.968357] LustreError: 6525:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a46785a5880 x1873624758735488/t266287972368(0) o36->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:551/0 lens 504/448 e 0 to 0 dl 1786832626 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4840.702976] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4846.564430] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4848.382871] Lustre: Failing over lustre-MDT0000 [ 4848.577230] Lustre: server umount lustre-MDT0000 complete [ 4869.155040] Lustre: 3652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786832585/real 1786832585] req@ffff8a467876ed80 x1873624772340992/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786832601 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4869.216882] Lustre: 3652:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4870.198709] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4870.200992] LDISKFS-fs (dm-0): recovery complete [ 4870.207307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4879.847166] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e8285bf330 [ 4884.021979] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4885.021326] Lustre: 6529:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a465ae13c50 x1873624758735744/t266287972369(0) o35->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:553/0 lens 392/456 e 0 to 0 dl 1786832628 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4885.046043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3297) [ 4885.047618] Lustre: 6529:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 4885.047929] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 4895.159891] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 18:23:47 (1786832627) [ 4896.237454] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4896.243333] Lustre: Skipped 1 previous similar message [ 4896.245465] LustreError: 18215:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a4646610000 x1873624758750080/t270582939664(0) o36->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:609/0 lens 504/448 e 0 to 0 dl 1786832684 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4896.257347] LustreError: 18215:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 4897.885960] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 4905.368345] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4907.102558] Lustre: Failing over lustre-MDT0000 [ 4907.315421] Lustre: server umount lustre-MDT0000 complete [ 4927.516602] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4927.518866] LDISKFS-fs (dm-0): recovery complete [ 4927.527347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4936.176078] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e8285bf98f [ 4936.320088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.324441] Lustre: Skipped 8 previous similar messages [ 4939.594056] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4941.847901] Lustre: 9388:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a4678390000 x1873624758750080/t270582939664(0) o36->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:654/0 lens 504/2880 e 0 to 0 dl 1786832729 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 4941.877374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3329) [ 4941.877407] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 4949.555304] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 18:24:42 (1786832682) [ 4950.632641] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 4952.288240] Lustre: *** cfs_fail_loc=13b, val=315*** [ 4952.290132] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 4952.297315] LustreError: 6528:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a465ae11050 x1873624758765056/t274877906960(0) o35->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:621/0 lens 392/456 e 0 to 0 dl 1786832696 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4960.016826] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4961.821652] Lustre: Failing over lustre-MDT0000 [ 4962.093190] Lustre: server umount lustre-MDT0000 complete [ 4982.920690] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4982.923171] LDISKFS-fs (dm-0): recovery complete [ 4982.929973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4992.897162] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4994.656317] Lustre: 40002:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a4644609880 x1873624758765056/t274877906960(0) o35->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:663/0 lens 392/456 e 0 to 0 dl 1786832738 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 4994.667762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 4994.668722] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 5004.803726] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 18:25:37 (1786832737) [ 5005.696406] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 5005.707538] LustreError: 14342:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a46442f0380 x1873624758777600/t279172874255(0) o101->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:718/0 lens 664/608 e 0 to 0 dl 1786832793 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5066.238837] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnecting [ 5066.253451] Lustre: Skipped 1 previous similar message [ 5066.280215] Lustre: 18215:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a464657d880 x1873624758777600/t279172874255(0) o101->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:24/0 lens 664/3488 e 0 to 0 dl 1786832854 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 5071.885204] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 18:26:44 (1786832804) [ 5078.573522] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5081.097227] Lustre: Failing over lustre-MDT0000 [ 5081.292768] Lustre: server umount lustre-MDT0000 complete [ 5081.568822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5081.584931] Lustre: Skipped 38 previous similar messages [ 5103.754743] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5103.758681] LDISKFS-fs (dm-0): recovery complete [ 5103.771657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.665456] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5117.934107] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5117.937063] Lustre: Skipped 44 previous similar messages [ 5117.992269] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3361) [ 5117.994554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 5125.567201] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5126.693714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5144.485804] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 18:27:57 (1786832877) [ 5152.071867] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5153.643438] Lustre: Failing over lustre-MDT0000 [ 5153.762113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5153.770670] Lustre: Skipped 4 previous similar messages [ 5153.959897] Lustre: server umount lustre-MDT0000 complete [ 5176.977794] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5176.979818] LDISKFS-fs (dm-0): recovery complete [ 5176.995350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5184.482063] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a467fe99f80 x1873624772512128/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5184.532775] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 5189.548347] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5190.213341] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 5190.218840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3425) [ 5198.725467] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5200.632982] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5206.626895] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 5218.328770] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 18:29:10 (1786832950) [ 5268.443783] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5270.171924] Lustre: Failing over lustre-MDT0000 [ 5270.699788] Lustre: server umount lustre-MDT0000 complete [ 5292.660780] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5292.662985] LDISKFS-fs (dm-0): recovery complete [ 5292.668949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5298.666618] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e8285d6606 [ 5303.994548] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5304.821730] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 5304.821873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 5313.377360] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5315.017194] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5375.769722] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 18:31:47 (1786833107) [ 5384.225737] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5385.951471] Lustre: Failing over lustre-MDT0000 [ 5386.171782] Lustre: server umount lustre-MDT0000 complete [ 5386.215179] LustreError: 18215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5386.232080] LustreError: 18215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 266 previous similar messages [ 5402.593124] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5402.603280] LustreError: Skipped 7 previous similar messages [ 5407.290343] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5407.295301] LDISKFS-fs (dm-0): recovery complete [ 5407.304760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.864053] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e828601931 [ 5413.224602] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5413.233555] Lustre: Skipped 8 previous similar messages [ 5417.325260] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5418.629964] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 5418.630730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 5426.028920] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5427.648526] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5437.166387] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 5439.126349] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 5446.313617] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 18:32:58 (1786833178) [ 5454.072057] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 5515.922395] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5515.928278] Lustre: Skipped 1 previous similar message [ 5515.930724] LustreError: 9388:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a46467df480 x1873624761589760/t296352743435(0) o36->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:473/0 lens 66040/440 e 0 to 0 dl 1786833303 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5576.203382] Lustre: 6527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a465ae43850 x1873624761589760/t296352743435(0) o36->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:534/0 lens 66040/440 e 0 to 0 dl 1786833364 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 5587.684594] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 5589.253075] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 18:35:21 (1786833321) [ 5603.374761] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5606.118548] Lustre: Failing over lustre-MDT0000 [ 5607.910693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5607.920122] LustreError: Skipped 7 previous similar messages [ 5608.438664] Lustre: server umount lustre-MDT0000 complete [ 5623.254399] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786833340/real 1786833340] req@ffff8a4677554e00 x1873624773377664/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786833356 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5623.304071] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 5630.868336] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5630.871320] LDISKFS-fs (dm-0): recovery complete [ 5630.878075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5633.056126] LustreError: 156121:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 5633.068375] LustreError: 156121:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8a46785b0a80 x1873624773383552/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1786833366 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5633.095464] LustreError: 156121:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 5633.516379] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e828604816 [ 5633.774900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5633.786972] Lustre: Skipped 5 previous similar messages [ 5634.191033] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5634.197161] Lustre: Skipped 7 previous similar messages [ 5638.120792] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5640.167196] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 5640.173858] Lustre: Skipped 7 previous similar messages [ 5640.205986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 5640.207542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 5647.406587] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5649.155673] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5660.731210] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 18:36:32 (1786833392) [ 5682.618990] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5694.291091] Lustre: Failing over lustre-OST0000 [ 5694.387805] Lustre: server umount lustre-OST0000 complete [ 5694.442276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5694.455222] Lustre: Skipped 19 previous similar messages [ 5716.391532] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5716.393724] LDISKFS-fs (dm-2): recovery complete [ 5716.399467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5718.132638] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5718.138728] Lustre: Skipped 22 previous similar messages [ 5721.942288] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5736.461613] Lustre: Failing over lustre-OST0000 [ 5736.623442] Lustre: server umount lustre-OST0000 complete [ 5754.657538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5760.497623] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5770.704423] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5772.976664] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5813.931897] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 18:39:06 (1786833546) [ 5816.590529] Lustre: Failing over lustre-MDT0000 [ 5817.007383] Lustre: server umount lustre-MDT0000 complete [ 5835.272055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.471976] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e828615614 [ 5849.417286] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5850.143150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 5850.145143] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 5863.088110] Lustre: Failing over lustre-MDT0000 [ 5863.295780] Lustre: server umount lustre-MDT0000 complete [ 5881.247511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5887.023436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 5887.023873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 5887.169117] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5895.198734] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5896.717379] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5903.999957] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 18:40:36 (1786833636) [ 5917.531019] Lustre: Failing over lustre-OST0000 [ 5917.604644] Lustre: server umount lustre-OST0000 complete [ 5934.452854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5939.815482] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5949.372982] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5950.615760] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5959.686240] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 18:41:32 (1786833692) [ 5961.460944] Lustre: Failing over lustre-MDT0000 [ 5961.709434] Lustre: server umount lustre-MDT0000 complete [ 5970.465852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.597877] Lustre: *** cfs_fail_loc=605, val=0*** [ 5970.600982] LustreError: 164849:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc09f84b0 failed: rc = -95 [ 5970.611296] LustreError: 164849:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 5970.617734] LustreError: 164849:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 5970.628768] LustreError: 164849:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 5970.633488] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 5970.645082] LustreError: 164849:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5970.656473] Lustre: server umount lustre-MDT0000 complete [ 5970.660143] LustreError: 164849:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 5979.158342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.443876] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5984.820859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 5984.821848] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 5990.681736] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 18:42:03 (1786833723) [ 5997.608034] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6000.232495] Lustre: Failing over lustre-MDT0000 [ 6000.444947] Lustre: server umount lustre-MDT0000 complete [ 6005.218366] LustreError: 135456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.236798] LustreError: 135456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 179 previous similar messages [ 6021.602522] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6021.613707] LustreError: Skipped 4 previous similar messages [ 6022.016804] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6022.018587] LDISKFS-fs (dm-0): recovery complete [ 6022.023619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6031.844241] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a46442f3800 x1873624773747968/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6031.867600] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 15 previous similar messages [ 6032.061408] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6032.069134] Lustre: Skipped 7 previous similar messages [ 6033.920848] Lustre: *** cfs_fail_loc=707, val=0*** [ 6036.659346] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6093.345706] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 6093.367046] Lustre: 167023:0:(ldlm_lib.c:2085:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 6093.371592] Lustre: 167023:0:(ldlm_lib.c:2085:extend_recovery_timer()) Skipped 1 previous similar message [ 6093.851620] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 6093.861300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 6099.726496] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6101.476897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6110.994046] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 18:44:03 (1786833843) [ 6140.638645] LustreError: 14342:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a464467c380 x1873624762493440/t0(0) o101->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:299/0 lens 664/0 e 0 to 0 dl 1786833884 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6140.663259] LustreError: 14342:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 6151.720689] LustreError: 14342:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6151.743851] LustreError: 6529:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a467fdc5880 x1873624762495104/t0(0) o35->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:310/0 lens 392/0 e 0 to 0 dl 1786833895 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6153.818485] LustreError: 6526:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a46442c9180 x1873624762500992/t0(0) o101->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:352/0 lens 576/0 e 0 to 0 dl 1786833937 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 6153.837434] LustreError: 6526:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 6157.805652] LustreError: 8919:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a46442f4000 x1873624773810432/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:316/0 lens 544/0 e 0 to 0 dl 1786833901 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 6157.833130] LustreError: 8919:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 28 previous similar messages [ 6168.429802] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 18:45:00 (1786833900) [ 6199.122152] LustreError: 43394:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 6210.176147] LustreError: 43394:0:(tgt_handler.c:2830:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 6223.029456] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 18:45:54 (1786833954) [ 6253.156351] LustreError: 18215:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a4644469f80 x1873624762520448/t0(0) o101->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:412/0 lens 576/0 e 0 to 0 dl 1786833997 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6253.186743] LustreError: 18215:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 6258.216147] LustreError: 18215:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6263.782892] LustreError: 43900:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a4543e00e00 x1873624773855872/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:422/0 lens 544/0 e 0 to 0 dl 1786834007 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 6263.808840] LustreError: 43900:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 6270.595264] LustreError: 14342:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6288.753687] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 18:47:01 (1786834021) [ 6393.389133] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 18:48:45 (1786834125) [ 6421.530860] LustreError: 9388:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a4646935180 x1873624762601344/t0(0) o101->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:580/0 lens 576/0 e 0 to 0 dl 1786834165 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 6421.570399] LustreError: 9388:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 99 previous similar messages [ 6421.578803] LustreError: 9388:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6421.585753] LustreError: 9388:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 6422.009112] LustreError: 9388:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6437.874949] LustreError: 40002:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 6437.887309] LustreError: 40002:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 6438.312145] LustreError: 40002:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 6438.321603] LustreError: 40002:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 6453.707050] LustreError: 6526:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff8a4646934700 x1873624762620288/t0(0) o36->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:612/0 lens 504/0 e 0 to 0 dl 1786834197 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 6453.733230] LustreError: 6526:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 78 previous similar messages [ 6471.147711] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 18:50:03 (1786834203) [ 6504.116470] Lustre: DEBUG MARKER: phase 2 [ 6512.651414] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 18:50:45 (1786834245) [ 6594.510374] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 18:52:06 (1786834326) [ 6595.887330] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 6597.328542] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 18:52:09 (1786834329) [ 6602.093304] Lustre: DEBUG MARKER: Started rundbench load pid=135322 ... [ 6611.828347] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6614.430752] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 6616.166803] Lustre: Failing over lustre-MDT0000 [ 6616.190495] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 6616.464428] Lustre: server umount lustre-MDT0000 complete [ 6617.616633] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6617.637418] LustreError: 6526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 31 previous similar messages [ 6621.159154] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6621.169598] Lustre: Skipped 23 previous similar messages [ 6621.172659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6621.177489] LustreError: Skipped 5 previous similar messages [ 6637.008137] Lustre: 3652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786834354/real 1786834354] req@ffff8a467882ad80 x1873624774061824/t0(0) o400->MGC192.168.206.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786834370 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6637.034627] Lustre: 3652:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6637.042532] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6637.538807] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 6637.545411] LDISKFS-fs (dm-0): recovery complete [ 6637.552470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6647.267487] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a467fe9b100 x1873624774070400/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6647.540560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.548075] Lustre: Skipped 7 previous similar messages [ 6647.571873] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6647.689239] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6647.694536] Lustre: Skipped 7 previous similar messages [ 6651.442162] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6652.903897] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6652.911958] Lustre: Skipped 22 previous similar messages [ 6653.576331] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 6653.593688] Lustre: Skipped 7 previous similar messages [ 6653.639294] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5046 to 0x2c0000401:5089) [ 6653.642069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5082 to 0x280000401:5121) [ 6661.042341] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6662.728671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6672.284777] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6675.015173] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 6676.696288] Lustre: Failing over lustre-MDT0001 [ 6676.959537] Lustre: server umount lustre-MDT0001 complete [ 6699.044839] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 6699.050406] LDISKFS-fs (dm-1): recovery complete [ 6699.070131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6703.680160] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6704.708662] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:404 to 0x2c0000400:449) [ 6704.708909] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:449) [ 6713.700749] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6715.292422] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6725.458951] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6728.040236] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 6729.849969] Lustre: Failing over lustre-MDT0000 [ 6729.883531] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 6730.093346] Lustre: server umount lustre-MDT0000 complete [ 6749.817163] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6749.822682] LDISKFS-fs (dm-0): recovery complete [ 6749.829861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6757.352057] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e828637ccf [ 6761.013564] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6763.456939] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5227 to 0x2c0000401:5249) [ 6763.457430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5258 to 0x280000401:5281) [ 6769.498347] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6771.064437] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6777.996630] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 18:55:10 (1786834510) [ 6905.861698] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6918.196244] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 6919.845564] Lustre: Failing over lustre-MDT0000 [ 6919.961231] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 6920.357859] Lustre: server umount lustre-MDT0000 complete [ 6942.134093] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6942.136226] LDISKFS-fs (dm-0): recovery complete [ 6942.142617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6947.315579] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e82867cc98 [ 6951.939613] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6956.838804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5550 to 0x2c0000401:5569) [ 6956.859519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5582 to 0x280000401:5601) [ 6962.479933] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6964.341475] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7064.880939] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 18:59:57 (1786834797) [ 7191.956339] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7204.079569] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 7205.635650] Lustre: Failing over lustre-MDT0000 [ 7205.656181] LustreError: 3654:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a4681889f80 x1873624778875648/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 7205.748939] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 7210.492317] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 7210.497587] Lustre: Skipped 5 previous similar messages [ 7211.176101] Lustre: server umount lustre-MDT0000 complete [ 7219.169286] LustreError: 9388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7219.186508] LustreError: 9388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 124 previous similar messages [ 7231.084133] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7231.090084] LDISKFS-fs (dm-0): recovery complete [ 7231.098632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.290306] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7248.040028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5865 to 0x280000401:5889) [ 7248.041172] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5833 to 0x2c0000401:5857) [ 7251.863149] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7252.988852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7258.824908] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 19:03:11 (1786834991) [ 7385.491568] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7397.262763] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 7398.595456] Lustre: Failing over lustre-MDT0001 [ 7398.675470] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -107 [ 7398.680497] LustreError: Skipped 3 previous similar messages [ 7398.684209] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7398.696071] Lustre: Skipped 16 previous similar messages [ 7398.700744] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7398.868112] Lustre: server umount lustre-MDT0001 complete [ 7417.300848] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 7417.303434] LDISKFS-fs (dm-1): recovery complete [ 7417.315389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7417.507708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7417.514385] Lustre: Skipped 4 previous similar messages [ 7417.540887] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7417.546801] Lustre: Skipped 4 previous similar messages [ 7419.203377] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7419.214790] Lustre: Skipped 4 previous similar messages [ 7420.598662] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7422.955344] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 7422.958892] Lustre: Skipped 20 previous similar messages [ 7422.975432] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7422.993431] Lustre: Skipped 4 previous similar messages [ 7423.021188] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1041 to 0x2c0000400:1057) [ 7423.023494] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1040 to 0x280000400:1057) [ 7427.841798] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7428.948191] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7556.192257] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7567.585342] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 7568.725879] Lustre: Failing over lustre-MDT0001 [ 7568.769756] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7568.894834] Lustre: server umount lustre-MDT0001 complete [ 7586.981289] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7586.983312] LDISKFS-fs (dm-1): recovery complete [ 7586.991592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.908793] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7592.475547] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1041 to 0x2c0000400:1089) [ 7592.476377] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1040 to 0x280000400:1089) [ 7596.683168] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7597.997289] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7604.302871] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 19:08:57 (1786835337) [ 7612.579991] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7614.509675] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 7615.662590] Lustre: Failing over lustre-OST0000 [ 7615.728984] Lustre: server umount lustre-OST0000 complete [ 7633.832937] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7633.835027] LDISKFS-fs (dm-2): recovery complete [ 7633.841604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7633.971527] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7638.450813] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7645.239792] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7646.474836] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7658.309549] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7660.550887] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 7662.071647] Lustre: Failing over lustre-OST0000 [ 7662.136638] Lustre: server umount lustre-OST0000 complete [ 7680.257787] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7680.259903] LDISKFS-fs (dm-2): recovery complete [ 7680.265913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7680.391941] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 7684.728812] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7692.240765] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7693.717760] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7702.183104] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 19:10:35 (1786835435) [ 7826.783979] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 7830.664825] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7841.716971] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 7842.731204] Lustre: Failing over lustre-MDT0001 [ 7842.748606] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7842.881062] Lustre: server umount lustre-MDT0001 complete [ 7844.353494] LustreError: 14342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7844.365815] LustreError: 14342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 79 previous similar messages [ 7844.751958] Lustre: Failing over lustre-MDT0000 [ 7851.202313] Lustre: server umount lustre-MDT0000 complete [ 7867.922123] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 7867.924399] LDISKFS-fs (dm-1): recovery complete [ 7867.934233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7867.978426] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7867.980990] LDISKFS-fs (dm-0): recovery complete [ 7867.988569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7868.072890] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7868.079546] LustreError: Skipped 3 previous similar messages [ 7868.082541] LustreError: 193319:0:(llog.c:1655:llog_backup()) MGC192.168.206.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7868.083646] LustreError: 193324:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.206.131@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff8a46560fbb00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7868.088876] Lustre: 193319:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7870.298733] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7870.428072] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7875.167390] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1041 to 0x2c0000400:1121) [ 7875.167747] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1040 to 0x280000400:1121) [ 7880.618213] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7071 to 0x2c0000401:7105) [ 7880.618415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7137) [ 7883.365594] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7884.130394] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7884.871484] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7889.871819] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 19:13:42 (1786835622) [ 7893.691531] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7895.071241] Lustre: Failing over lustre-MDT0000 [ 7895.259392] Lustre: server umount lustre-MDT0000 complete [ 7912.855295] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 7912.857657] LDISKFS-fs (dm-0): recovery complete [ 7912.865559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7915.820332] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7915.972505] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 7931.407902] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7931.475387] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7107 to 0x2c0000401:7137) [ 7931.477019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7169) [ 7934.856462] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7935.630473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7940.846547] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 19:14:33 (1786835673) [ 7945.348633] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7946.918214] Lustre: Failing over lustre-MDT0000 [ 7947.124735] Lustre: server umount lustre-MDT0000 complete [ 7964.591351] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7964.593555] LDISKFS-fs (dm-0): recovery complete [ 7964.599345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7964.678530] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7964.687139] LustreError: Skipped 1 previous similar message [ 7967.213219] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7968.213683] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 7968.216957] LustreError: 193331:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a46780fe300 x1873624780226816/t335007461798(335007461798) o101->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:617/0 lens 520/664 e 0 to 0 dl 1786835712 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7983.629965] Lustre: lustre-MDT0000: Client 0c1246a1-a3e7-41e8-83f3-2e40bbd471be (at 192.168.206.31@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7983.647293] Lustre: 193331:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a46465d9f80 x1873624780226816/t335007461798(335007461798) o101->0c1246a1-a3e7-41e8-83f3-2e40bbd471be@192.168.206.31@tcp:632/0 lens 520/3488 e 0 to 0 dl 1786835727 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 7983.726578] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7139 to 0x2c0000401:7169) [ 7983.727246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7201) [ 7986.991262] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7987.840680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7993.163870] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 19:15:26 (1786835726) [ 7995.566862] Lustre: Failing over lustre-OST0000 [ 7995.639350] Lustre: server umount lustre-OST0000 complete [ 7997.758994] Lustre: Failing over lustre-MDT0000 [ 7997.896307] Lustre: server umount lustre-MDT0000 complete [ 7998.948073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7998.953725] LustreError: Skipped 6 previous similar messages [ 7998.956413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7998.965883] Lustre: Skipped 24 previous similar messages [ 8012.737823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8015.611413] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8017.061301] Lustre: lustre-MDT0000: Denying connection for new client efb8c570-945d-4db7-b293-126bb04c4f7b (at 192.168.206.31@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 8018.440576] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7139 to 0x2c0000401:7201) [ 8025.862028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8025.967855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.971587] Lustre: Skipped 6 previous similar messages [ 8025.977076] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 8025.985523] Lustre: Skipped 8 previous similar messages [ 8027.020413] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8027.025312] Lustre: Skipped 8 previous similar messages [ 8029.660020] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8031.209521] Lustre: lustre-OST0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8031.209707] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8031.214102] Lustre: Skipped 8 previous similar messages [ 8031.219335] Lustre: Skipped 26 previous similar messages [ 8031.230836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7233) [ 8036.680556] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 19:16:09 (1786835769) [ 8037.277267] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8037.279721] LustreError: 193337:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a4646796680 x1873624785516928/t352187318279(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:686/0 lens 1312/4320 e 0 to 0 dl 1786835781 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8041.079881] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8041.975364] Lustre: Failing over lustre-MDT0000 [ 8042.175718] Lustre: server umount lustre-MDT0000 complete [ 8053.731534] Lustre: 193357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786835770/real 1786835770] req@ffff8a4646794e00 x1873624785516928/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1786835786 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8053.747966] Lustre: 193357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 8058.729962] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8058.732957] LDISKFS-fs (dm-0): recovery complete [ 8058.742985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8061.470953] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8064.546752] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7233) [ 8064.546752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7265) [ 8067.710433] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8068.656571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8074.372052] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 19:16:47 (1786835807) [ 8075.005277] LustreError: 194114:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a46455b6a00 x1873624785549312/t356482285580(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:724/0 lens 2520/4320 e 0 to 0 dl 1786835819 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8078.886801] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8083.009503] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8084.041776] Lustre: Failing over lustre-MDT0001 [ 8084.064226] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.31@tcp (stopping) [ 8084.066649] Lustre: Skipped 4 previous similar messages [ 8090.253782] Lustre: server umount lustre-MDT0001 complete [ 8106.517485] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8106.519776] LDISKFS-fs (dm-1): recovery complete [ 8106.531728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.715562] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8112.132536] Lustre: 193332:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a464436ed80 x1873624782946560/t21474837917(0) o36->efb8c570-945d-4db7-b293-126bb04c4f7b@192.168.206.31@tcp:32/0 lens 560/2880 e 0 to 0 dl 1786835882 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8112.139091] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1132 to 0x280000400:1153) [ 8112.139161] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1132 to 0x2c0000400:1153) [ 8115.340280] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8116.240750] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8121.652837] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 19:17:34 (1786835854) [ 8122.296126] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8122.298137] Lustre: Skipped 1 previous similar message [ 8126.476421] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8130.642773] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8131.799395] Lustre: Failing over lustre-MDT0000 [ 8132.081779] Lustre: server umount lustre-MDT0000 complete [ 8138.720457] Lustre: 204197:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786835855/real 1786835855] req@ffff8a4679b7a680 x1873624785583104/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1786835871 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8138.736985] Lustre: 204197:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8148.960352] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8148.969586] LustreError: Skipped 2 previous similar messages [ 8149.015916] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8149.020584] LDISKFS-fs (dm-0): recovery complete [ 8149.026445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8158.178092] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a4552d8d880 x1873624785600128/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8158.191708] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 57 previous similar messages [ 8160.871841] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8163.972079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7297) [ 8163.972531] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7265) [ 8167.092844] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8168.163882] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8170.810818] Lustre: Failing over lustre-MDT0001 [ 8170.994882] Lustre: server umount lustre-MDT0001 complete [ 8188.258646] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8188.260892] LDISKFS-fs (dm-1): recovery complete [ 8188.270407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8190.736755] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8193.548207] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1164 to 0x2c0000400:1185) [ 8193.551301] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1164 to 0x280000400:1185) [ 8196.664730] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8197.599970] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8203.609819] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 19:18:56 (1786835936) [ 8204.309643] LustreError: 193336:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a4553b1a680 x1873624785646336/t360777252880(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:98/0 lens 2520/4320 e 0 to 0 dl 1786835948 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8204.332825] LustreError: 193336:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 1 previous similar message [ 8211.536387] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8215.510405] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8216.512028] Lustre: Failing over lustre-MDT0000 [ 8216.752468] Lustre: server umount lustre-MDT0000 complete [ 8218.580462] LustreError: 43908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786835951 with bad export cookie 13299527620434503342 [ 8218.583646] Lustre: Failing over lustre-MDT0001 [ 8218.590261] LustreError: 43908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8218.595121] LustreError: 209840:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x2000284a1:0xa:0x0].0x0 (ffff8a45509f1c00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8218.604583] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.31@tcp (stopping) [ 8218.612613] Lustre: Skipped 7 previous similar messages [ 8223.712919] LustreError: 209840:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a46817fb100 x1873624785658112/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8223.725479] LustreError: 209840:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8223.881892] Lustre: server umount lustre-MDT0001 complete [ 8240.193671] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8240.198971] LDISKFS-fs (dm-0): recovery complete [ 8240.207910] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8240.208457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8240.210431] LDISKFS-fs (dm-1): recovery complete [ 8240.219982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8240.332458] LustreError: 210941:0:(llog.c:1655:llog_backup()) MGC192.168.206.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8240.336618] Lustre: 210941:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8243.680831] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a4678775f80 x1873624785659264/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8243.697668] Lustre: MGS: Client 966d54c7-b640-49a0-a4a9-60a5e22fc676 (at 0@lo) reconnecting [ 8243.705681] Lustre: Skipped 2 previous similar messages [ 8246.025996] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8246.190131] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8248.991251] Lustre: 211398:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a46447ad180 x1873624783017472/t30064771142(0) o36->efb8c570-945d-4db7-b293-126bb04c4f7b@192.168.206.31@tcp:167/0 lens 560/2880 e 0 to 0 dl 1786836017 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8248.995602] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1196 to 0x2c0000400:1217) [ 8248.995606] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1196 to 0x280000400:1217) [ 8249.041725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7329) [ 8249.041746] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7297) [ 8252.073418] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.964591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8253.725519] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8258.696499] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 19:19:51 (1786835991) [ 8259.237443] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 8259.239986] Lustre: Skipped 1 previous similar message [ 8266.223384] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8267.284719] Lustre: Failing over lustre-MDT0000 [ 8267.421497] Lustre: server umount lustre-MDT0000 complete [ 8274.434286] Lustre: 210949:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a4681b7ea00 x1873624783051648/t34359738438(0) o36->efb8c570-945d-4db7-b293-126bb04c4f7b@192.168.206.31@tcp:168/0 lens 560/2880 e 0 to 0 dl 1786836018 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8283.946156] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8283.948308] LDISKFS-fs (dm-0): recovery complete [ 8283.952606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8286.453879] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8289.311140] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7329) [ 8289.311209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7361) [ 8292.042181] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8292.731585] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8297.847405] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 19:20:30 (1786836030) [ 8302.394737] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8303.404642] Lustre: Failing over lustre-MDT0001 [ 8303.506228] Lustre: server umount lustre-MDT0001 complete [ 8319.572980] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8319.574697] LDISKFS-fs (dm-1): recovery complete [ 8319.584410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8322.046787] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8325.122849] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1238 to 0x280000400:1281) [ 8325.123561] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1238 to 0x2c0000400:1281) [ 8325.124176] Lustre: 211398:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a4678654000 x1873624783083648/t34359738509(0) o36->efb8c570-945d-4db7-b293-126bb04c4f7b@192.168.206.31@tcp:219/0 lens 560/2880 e 0 to 0 dl 1786836069 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8328.022206] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8328.952809] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8334.138701] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 19:21:07 (1786836067) [ 8334.660714] LustreError: 210949:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a4553b1b100 x1873624783113600/t38654705734(0) o36->efb8c570-945d-4db7-b293-126bb04c4f7b@192.168.206.31@tcp:228/0 lens 560/448 e 0 to 0 dl 1786836078 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8334.674532] LustreError: 210949:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 2 previous similar messages [ 8341.433098] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8345.324885] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8346.241986] Lustre: Failing over lustre-MDT0000 [ 8346.470172] Lustre: server umount lustre-MDT0000 complete [ 8351.230582] Lustre: lustre-MDT0001: Client efb8c570-945d-4db7-b293-126bb04c4f7b (at 192.168.206.31@tcp) reconnecting [ 8351.236593] Lustre: Skipped 1 previous similar message [ 8362.681408] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8362.683160] LDISKFS-fs (dm-0): recovery complete [ 8362.694869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.092757] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8368.171966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7393) [ 8368.172468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7361) [ 8370.846369] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8371.577729] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8373.897209] Lustre: Failing over lustre-MDT0001 [ 8374.018368] Lustre: server umount lustre-MDT0001 complete [ 8390.337672] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8390.340766] LDISKFS-fs (dm-1): recovery complete [ 8390.364254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8392.751786] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8395.781786] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1292 to 0x2c0000400:1313) [ 8395.781786] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1292 to 0x280000400:1313) [ 8398.548715] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8399.415924] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8404.893113] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 19:22:17 (1786836137) [ 8412.588216] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8416.486303] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8417.483492] Lustre: Failing over lustre-MDT0000 [ 8417.704317] Lustre: server umount lustre-MDT0000 complete [ 8419.633916] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836152 with bad export cookie 13299527620434517153 [ 8419.636281] Lustre: Failing over lustre-MDT0001 [ 8419.639423] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8419.789313] Lustre: server umount lustre-MDT0001 complete [ 8437.120737] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8437.120754] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8437.123029] LDISKFS-fs (dm-1): recovery complete [ 8437.128343] LDISKFS-fs (dm-0): recovery complete [ 8437.133886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.137050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.217171] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786836154/real 1786836154] req@ffff8a46815ced80 x1873624785824896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786836170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8437.229876] Lustre: 3651:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 8437.276478] LustreError: 222091:0:(llog.c:1655:llog_backup()) MGC192.168.206.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8437.281024] Lustre: 222091:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8444.384586] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a467fb9a300 x1873624785828096/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8444.394434] LustreError: 222110:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8444.408604] LustreError: 222110:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 216 previous similar messages [ 8446.436236] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8446.587495] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8447.841471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7425) [ 8447.844212] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7393) [ 8447.855716] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1324 to 0x280000400:1345) [ 8447.855783] Lustre: 222110:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a45533ded80 x1873624783153664/t42949673030(0) o36->efb8c570-945d-4db7-b293-126bb04c4f7b@192.168.206.31@tcp:341/0 lens 560/2880 e 0 to 0 dl 1786836191 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 8447.857410] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1324 to 0x2c0000400:1345) [ 8447.877321] Lustre: 222110:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8451.305553] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8452.102725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8452.819472] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8458.074758] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 19:23:11 (1786836191) [ 8462.483301] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8463.460438] Lustre: Failing over lustre-MDT0001 [ 8463.464291] LustreError: 223899:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x2000284a1:0x17:0x0].0xf7117594 (ffff8a467878a500) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 8469.054928] Lustre: server umount lustre-MDT0001 complete [ 8484.363625] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8484.365413] LDISKFS-fs (dm-1): recovery complete [ 8484.371891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8486.454733] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8489.980883] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1356 to 0x280000400:1377) [ 8489.984676] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1356 to 0x2c0000400:1377) [ 8492.668492] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8493.455843] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8498.082438] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 19:23:51 (1786836231) [ 8502.404909] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8503.407517] Lustre: Failing over lustre-MDT0000 [ 8503.699145] Lustre: server umount lustre-MDT0000 complete [ 8519.553792] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8519.555526] LDISKFS-fs (dm-0): recovery complete [ 8519.559658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8519.643650] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8519.649686] LustreError: Skipped 5 previous similar messages [ 8521.602946] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8524.812518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7457) [ 8524.812625] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7425) [ 8527.449804] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8528.229910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8532.531553] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 19:24:25 (1786836265) [ 8533.028960] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 8533.031040] Lustre: Skipped 5 previous similar messages [ 8536.022098] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8539.017252] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8539.731886] Lustre: Failing over lustre-MDT0000 [ 8540.004529] Lustre: server umount lustre-MDT0000 complete [ 8555.170353] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8555.172312] LDISKFS-fs (dm-0): recovery complete [ 8555.176818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8557.201449] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8560.661673] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7457) [ 8560.661906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7489) [ 8563.236736] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8563.943115] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8566.175932] Lustre: Failing over lustre-MDT0001 [ 8566.269774] Lustre: server umount lustre-MDT0001 complete [ 8581.292153] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8581.294858] LDISKFS-fs (dm-1): recovery complete [ 8581.302786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.191985] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8586.748004] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1356 to 0x2c0000400:1409) [ 8586.748925] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1356 to 0x280000400:1409) [ 8589.084609] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8589.841237] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8594.151869] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 19:25:27 (1786836327) [ 8594.655734] LustreError: 8449:0:(ldlm_lib.c:3346:target_send_reply_msg()) @@@ dropping reply req@ffff8a46817f8e00 x1873624785964672/t386547056654(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:488/0 lens 1728/4320 e 0 to 0 dl 1786836338 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 8594.665839] LustreError: 8449:0:(ldlm_lib.c:3346:target_send_reply_msg()) Skipped 4 previous similar messages [ 8597.925554] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8601.128892] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8601.869122] Lustre: Failing over lustre-MDT0000 [ 8602.080502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8602.083561] LustreError: Skipped 12 previous similar messages [ 8602.085198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8602.090536] Lustre: Skipped 52 previous similar messages [ 8602.092720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8602.096289] Lustre: Skipped 10 previous similar messages [ 8602.159982] Lustre: server umount lustre-MDT0000 complete [ 8603.647160] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836336 with bad export cookie 13299527620434527625 [ 8603.649289] Lustre: Failing over lustre-MDT0001 [ 8603.655732] LustreError: 6510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8608.737022] LustreError: 232144:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8a4545a8dc00 x1873624785972224/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 8608.747205] LustreError: 232144:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 8608.863902] Lustre: server umount lustre-MDT0001 complete [ 8624.644940] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8624.647577] LDISKFS-fs (dm-0): recovery complete [ 8624.653441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8624.665412] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8624.668778] LDISKFS-fs (dm-1): recovery complete [ 8624.679344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8628.849919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8628.853566] Lustre: Skipped 16 previous similar messages [ 8628.882206] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8628.887657] Lustre: Skipped 16 previous similar messages [ 8630.799996] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8630.873742] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8633.121114] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8633.122616] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8633.124477] Lustre: Skipped 17 previous similar messages [ 8633.127624] Lustre: Skipped 56 previous similar messages [ 8634.039560] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 8634.043090] Lustre: Skipped 16 previous similar messages [ 8634.069848] Lustre: 233259:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8a4679082a00 x1873624783234688/t55834574850(0) o36->efb8c570-945d-4db7-b293-126bb04c4f7b@192.168.206.31@tcp:554/0 lens 496/2888 e 0 to 0 dl 1786836404 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 8634.081086] Lustre: 233259:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 8634.086993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7521) [ 8634.087259] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7489) [ 8634.098251] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1356 to 0x2c0000400:1441) [ 8634.098251] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1356 to 0x280000400:1441) [ 8636.331276] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8636.987665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8637.581562] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8641.608347] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 19:26:14 (1786836374) [ 8645.173980] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8645.893268] Lustre: Failing over lustre-MDT0000 [ 8646.068293] Lustre: server umount lustre-MDT0000 complete [ 8657.403331] Lustre: lustre-MDT0001: Client efb8c570-945d-4db7-b293-126bb04c4f7b (at 192.168.206.31@tcp) reconnecting [ 8660.229207] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8660.230481] LDISKFS-fs (dm-0): recovery complete [ 8660.234796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8661.762083] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8665.604647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7553) [ 8665.604720] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7521) [ 8667.748411] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8668.310620] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8672.000233] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 19:26:45 (1786836405) [ 8675.218909] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8675.875229] Lustre: Failing over lustre-MDT0001 [ 8675.955970] Lustre: server umount lustre-MDT0001 complete [ 8690.433124] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8690.434937] LDISKFS-fs (dm-1): recovery complete [ 8690.440606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8692.192223] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8695.798089] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1356 to 0x280000400:1473) [ 8695.798203] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1356 to 0x2c0000400:1473) [ 8697.917609] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8698.507994] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8702.386753] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 19:27:15 (1786836435) [ 8705.717264] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8708.596105] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8709.321505] Lustre: Failing over lustre-MDT0000 [ 8709.536654] Lustre: server umount lustre-MDT0000 complete [ 8724.045904] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8724.047778] LDISKFS-fs (dm-0): recovery complete [ 8724.051931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8725.672590] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8729.599264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7585) [ 8729.599276] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7553) [ 8731.527354] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8732.069311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8734.032451] Lustre: Failing over lustre-MDT0001 [ 8734.119648] Lustre: server umount lustre-MDT0001 complete [ 8748.767236] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8748.771854] LDISKFS-fs (dm-1): recovery complete [ 8748.783111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8750.473567] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8754.168100] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1356 to 0x2c0000400:1505) [ 8754.168655] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1356 to 0x280000400:1505) [ 8756.601962] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8757.282450] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8761.238633] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 19:28:14 (1786836494) [ 8764.722865] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8767.712826] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8768.510914] Lustre: Failing over lustre-MDT0000 [ 8768.691606] Lustre: server umount lustre-MDT0000 complete [ 8770.139423] LustreError: 43950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836503 with bad export cookie 13299527620434535199 [ 8770.139947] Lustre: Failing over lustre-MDT0001 [ 8770.144823] LustreError: 43950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8770.272989] Lustre: server umount lustre-MDT0001 complete [ 8785.741589] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8785.743897] LDISKFS-fs (dm-0): recovery complete [ 8785.748670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.750454] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8785.754821] LDISKFS-fs (dm-1): recovery complete [ 8785.760544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8791.008262] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786836508/real 1786836508] req@ffff8a4678288e00 x1873624786107008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786836524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8791.022403] Lustre: 3653:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [ 8795.104601] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8a46817f9180 x1873624786108928/t0(0) o250->MGC192.168.206.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8795.116115] LustreError: 3650:0:(client.c:1391:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 8797.056425] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8797.241281] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8802.769665] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1356 to 0x280000400:1537) [ 8802.769665] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1356 to 0x2c0000400:1537) [ 8809.963071] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7203 to 0x2c0000401:7585) [ 8809.963151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7103 to 0x280000401:7617) [ 8812.393799] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8813.050796] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8813.661041] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8817.881421] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 19:29:11 (1786836551) [ 8818.686361] Lustre: 245821:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting efb8c570-945d-4db7-b293-126bb04c4f7b at adminstrative request [ 8824.300406] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 19:29:17 (1786836557) [ 8827.495968] Lustre: Failing over lustre-MDT0000 [ 8827.682488] Lustre: server umount lustre-MDT0000 complete [ 8841.245858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8843.092307] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8846.878744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7637 to 0x2c0000401:7681) [ 8846.878766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7668 to 0x280000401:7713) [ 8848.875035] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8849.440625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8853.077368] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 19:29:46 (1786836586) [ 8858.445283] Lustre: Failing over lustre-OST0000 [ 8858.514942] Lustre: server umount lustre-OST0000 complete [ 8871.508924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8873.607887] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8877.667113] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8878.228951] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8882.035959] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 19:30:15 (1786836615) [ 8883.482412] Lustre: Failing over lustre-MDT0000 [ 8883.690224] Lustre: server umount lustre-MDT0000 complete [ 8884.708050] LustreError: 43908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836617 with bad export cookie 13299527620434547883 [ 8886.425077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8888.213019] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8889.064738] Lustre: lustre-MDT0000: Denying connection for new client 92568bde-2684-4416-8645-bf2f38f7748a (at 192.168.206.31@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 8889.071905] Lustre: Skipped 2 previous similar messages [ 8891.897886] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7637 to 0x2c0000401:7713) [ 8891.898652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7668 to 0x280000401:7745) [ 8896.735520] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 19:30:29 (1786836629) [ 8899.545785] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8900.428661] Lustre: Failing over lustre-OST0000 [ 8900.464301] Lustre: server umount lustre-OST0000 complete [ 8915.007164] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8915.008718] LDISKFS-fs (dm-2): recovery complete [ 8915.013964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8917.440104] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8921.643367] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8922.289643] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8926.182882] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 19:30:59 (1786836659) [ 8929.468221] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8931.558277] Lustre: Failing over lustre-OST0000 [ 8931.598947] Lustre: server umount lustre-OST0000 complete [ 8947.022608] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 8947.024610] LDISKFS-fs (dm-2): recovery complete [ 8947.033740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8949.118191] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.206.31@tcp inode [0x20002f201:0x5:0x0] object 0x280000401:7747 extent [0-4194303]: client csum 5cd09410, server csum b3512169 [ 8949.570183] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8953.668256] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8954.231495] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8958.056828] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 19:31:31 (1786836691) [ 8960.962520] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8964.105864] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8966.402936] Lustre: Failing over lustre-MDT0000 [ 8966.578361] Lustre: server umount lustre-MDT0000 complete [ 8967.919969] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786836700 with bad export cookie 13299527620434562107 [ 8967.922066] Lustre: Failing over lustre-OST0000 [ 8967.924470] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8967.978641] Lustre: server umount lustre-OST0000 complete [ 8982.810625] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8982.812286] LDISKFS-fs (dm-0): recovery complete [ 8982.816220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8992.225385] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb89169e8287f9b5a [ 8993.746859] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8997.464230] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7637 to 0x2c0000401:7745) [ 9008.051871] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9008.054580] LDISKFS-fs (dm-2): recovery complete [ 9008.058590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9009.948631] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9017.068493] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 19:32:30 (1786836750) [ 9022.010879] Lustre: Failing over lustre-OST0000 [ 9022.072667] Lustre: server umount lustre-OST0000 complete [ 9023.479783] Lustre: Failing over lustre-MDT0000 [ 9023.825952] Lustre: server umount lustre-MDT0000 complete [ 9037.023882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9038.673191] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9042.428913] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7637 to 0x2c0000401:7777) [ 9044.987373] LustreError: 43906:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9044.995058] LustreError: 43906:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 233 previous similar messages [ 9047.093632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9049.226694] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9050.005698] Lustre: lustre-OST0000: Denying connection for new client 1fe65f11-9e06-44aa-a893-620ecd221e60 (at 192.168.206.31@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 9055.227424] Lustre: lustre-OST0000: Denying connection for new client 1fe65f11-9e06-44aa-a893-620ecd221e60 (at 192.168.206.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 9065.467575] Lustre: lustre-OST0000: Denying connection for new client 1fe65f11-9e06-44aa-a893-620ecd221e60 (at 192.168.206.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 9065.472494] Lustre: Skipped 1 previous similar message [ 9085.947671] Lustre: lustre-OST0000: Denying connection for new client 1fe65f11-9e06-44aa-a893-620ecd221e60 (at 192.168.206.31@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 9085.952712] Lustre: Skipped 3 previous similar messages [ 9118.500118] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 9118.503224] Lustre: 259357:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 92568bde-2684-4416-8645-bf2f38f7748a@ [ 9118.508379] Lustre: 259357:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 9118.511829] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 9118.523331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7796 to 0x280000401:7817) [ 9122.585162] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 9127.159153] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 9129.347136] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 19:34:22 (1786836862) [ 9130.797150] Lustre: Failing over lustre-OST0000 [ 9130.841887] Lustre: server umount lustre-OST0000 complete [ 9144.048807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9146.136272] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9150.608636] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 19:34:43 (1786836883) [ 9152.018813] Lustre: Failing over lustre-OST0000 [ 9152.508889] Lustre: lustre-OST0000: Not available for connect from 192.168.206.31@tcp (stopping) [ 9152.511540] Lustre: Skipped 6 previous similar messages [ 9154.115290] Lustre: server umount lustre-OST0000 complete [ 9167.416654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9168.679127] LustreError: 262363:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 9168.683071] LustreError: 262363:0:(ldlm_lib.c:2905:target_recovery_thread()) Skipped 77 previous similar messages [ 9169.755420] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9175.008147] Lustre: *** cfs_fail_loc=715, val=40*** [ 9184.224765] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:25 [ 9184.228475] Lustre: Skipped 1 previous similar message [ 9190.368125] Lustre: *** cfs_fail_loc=715, val=40*** [ 9190.370088] Lustre: Skipped 2 previous similar messages [ 9200.608654] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 9200.612884] Lustre: Skipped 2 previous similar messages [ 9206.752223] Lustre: *** cfs_fail_loc=715, val=40*** [ 9206.753837] Lustre: Skipped 2 previous similar messages [ 9208.728104] LustreError: 262363:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9208.731113] LustreError: 262363:0:(ldlm_lib.c:2905:target_recovery_thread()) Skipped 77 previous similar messages [ 9210.783487] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9211.312053] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9214.803388] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 19:35:48 (1786836948) [ 9216.139339] Lustre: Failing over lustre-MDT0000 [ 9216.480408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9216.485071] LustreError: Skipped 17 previous similar messages [ 9216.486288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9216.490949] Lustre: Skipped 53 previous similar messages [ 9216.500322] Lustre: server umount lustre-MDT0000 complete [ 9229.423904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9229.485358] LustreError: MGC192.168.206.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9229.489568] LustreError: Skipped 9 previous similar messages [ 9229.565795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9229.568282] Lustre: Skipped 18 previous similar messages [ 9229.596047] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9229.597952] Lustre: Skipped 18 previous similar messages [ 9231.057411] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9234.914117] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9234.915989] Lustre: Skipped 54 previous similar messages [ 9234.924909] LustreError: 263914:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 9237.472301] Lustre: *** cfs_fail_loc=715, val=80*** [ 9237.474018] Lustre: Skipped 2 previous similar messages [ 9247.739070] Lustre: lustre-MDT0000: Client 1fe65f11-9e06-44aa-a893-620ecd221e60 (at 192.168.206.31@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 9247.742540] Lustre: Skipped 1 previous similar message [ 9250.272594] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9253.856314] Lustre: *** cfs_fail_loc=715, val=80*** [ 9253.857911] Lustre: Skipped 1 previous similar message [ 9264.123397] Lustre: lustre-MDT0000: Client 1fe65f11-9e06-44aa-a893-620ecd221e60 (at 192.168.206.31@tcp) reconnected, waiting for 2 clients in recovery for 1:11 [ 9266.656463] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9270.240150] Lustre: *** cfs_fail_loc=715, val=80*** [ 9270.241761] Lustre: Skipped 1 previous similar message [ 9279.483525] Lustre: lustre-MDT0000: Client 1fe65f11-9e06-44aa-a893-620ecd221e60 (at 192.168.206.31@tcp) reconnected, waiting for 2 clients in recovery for 0:56 [ 9282.016980] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9288.160413] Lustre: *** cfs_fail_loc=715, val=80*** [ 9288.164110] Lustre: Skipped 2 previous similar messages [ 9295.867254] Lustre: lustre-MDT0000: Client 1fe65f11-9e06-44aa-a893-620ecd221e60 (at 192.168.206.31@tcp) reconnected, waiting for 2 clients in recovery for 0:39 [ 9298.400560] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9312.251148] Lustre: lustre-MDT0000: Client 1fe65f11-9e06-44aa-a893-620ecd221e60 (at 192.168.206.31@tcp) reconnected, waiting for 2 clients in recovery for 0:23 [ 9314.784624] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 9315.008089] LustreError: 263914:0:(ldlm_lib.c:2905:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 9315.014299] Lustre: 263914:0:(ldlm_lib.c:2951:target_recovery_thread()) too long recovery - read logs [ 9315.017279] LustreError: dumping log to /tmp/lustre-log.1786837048.263914 [ 9315.063987] Lustre: lustre-MDT0000: Recovery over after 1:24, of 2 clients 2 recovered and 0 were evicted. [ 9315.067087] Lustre: Skipped 18 previous similar messages [ 9315.083412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7831 to 0x280000401:7849) [ 9315.083418] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7790 to 0x2c0000401:7809) [ 9317.031034] Lustre: DEBUG MARKER: oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9317.570665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9321.231217] Lustre: DEBUG MARKER: == replay-single test complete, duration 9053 sec ======== 19:37:34 (1786837054) [ 9321.837517] Lustre: DEBUG MARKER: === replay-single: start cleanup 19:37:35 (1786837055) === [ 9324.872800] Lustre: DEBUG MARKER: === replay-single: finish cleanup 19:37:38 (1786837058) === [ 9361.560576] Lustre: server umount lustre-MDT0000 complete [ 9364.317709] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786837097 with bad export cookie 13299527620434574847 [ 9364.322115] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9364.440407] Lustre: server umount lustre-MDT0001 complete [ 9377.662324] Lustre: server umount lustre-OST0000 complete [ 9380.530035] Lustre: server umount lustre-OST0001 complete [ 9386.144767] Lustre: DEBUG MARKER: oleg631-server.virtnet: executing unload_modules_local [ 9387.177375] Key type lgssc unregistered [ 9387.323438] LNet: 267206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9387.327247] LNetError: 267206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9387.342596] LNet: Removed LNI 192.168.206.131@tcp [ 9387.671118] Key type .llcrypt unregistered [ 9387.672339] Key type ._llcrypt unregistered