[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 747575221 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002348] x2apic enabled [ 0.003021] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.008789] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010023] pid_max: default: 32768 minimum: 301 [ 0.011156] LSM: Security Framework initializing [ 0.012052] Yama: becoming mindful. [ 0.013000] SELinux: Initializing. [ 0.013000] *** VALIDATE selinux *** [ 0.020052] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025088] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027094] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029003] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030000] *** VALIDATE tmpfs *** [ 0.031089] *** VALIDATE proc *** [ 0.032260] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.035057] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036186] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038034] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042636] debug: unmapping init [mem 0xffffffff92c59000-0xffffffff92c60fff] [ 0.044883] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045773] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047015] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049020] ... value mask: 0000ffffffffffff [ 0.050023] ... max period: 00007fffffffffff [ 0.051019] ... fixed-purpose events: 3 [ 0.052017] ... event mask: 000000070000000f [ 0.053408] rcu: Hierarchical SRCU implementation. [ 0.055585] smp: Bringing up secondary CPUs ... [ 0.056595] x86: Booting SMP configuration: [ 0.057020] .... node #0, CPUs: #1 #2 #3 [ 0.064413] smp: Brought up 1 node, 4 CPUs [ 0.066015] smpboot: Max logical packages: 1 [ 0.067014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.102554] node 0 deferred pages initialised in 31ms [ 0.104392] devtmpfs: initialized [ 0.105275] x86/mm: Memory block size: 128MB [ 0.108111] gcov: version magic: 0x41383552 [ 0.110926] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.111093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.112423] pinctrl core: initialized pinctrl subsystem [ 0.113371] [ 0.113903] ************************************************************* [ 0.114014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.115011] ** ** [ 0.116011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.117011] ** ** [ 0.118012] ** This means that this kernel is built to expose internal ** [ 0.119013] ** IOMMU data structures, which may compromise security on ** [ 0.120012] ** your system. ** [ 0.121013] ** ** [ 0.122010] ** If you see this message and you are not debugging the ** [ 0.123010] ** kernel, report this immediately to your vendor! ** [ 0.124012] ** ** [ 0.125167] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.126012] ************************************************************* [ 0.127805] NET: Registered protocol family 16 [ 0.128598] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.129089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.130082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.133000] cpuidle: using governor menu [ 0.134000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.137676] PCI: Using configuration type 1 for base access [ 0.140156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.149168] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.152744] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.155225] cryptd: max_cpu_qlen set to 1000 [ 0.159000] ACPI: Added _OSI(Module Device) [ 0.161010] ACPI: Added _OSI(Processor Device) [ 0.163012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.165015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.170087] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.176619] ACPI: Interpreter enabled [ 0.178060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.179010] ACPI: Using IOAPIC for interrupt routing [ 0.180096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.183407] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.193000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.195040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.198019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.201101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.206259] acpiphp: Slot [2] registered [ 0.208242] acpiphp: Slot [3] registered [ 0.209101] acpiphp: Slot [4] registered [ 0.211087] acpiphp: Slot [5] registered [ 0.212121] acpiphp: Slot [6] registered [ 0.214131] acpiphp: Slot [7] registered [ 0.217040] acpiphp: Slot [8] registered [ 0.219328] acpiphp: Slot [9] registered [ 0.227187] acpiphp: Slot [10] registered [ 0.229106] acpiphp: Slot [11] registered [ 0.230097] acpiphp: Slot [12] registered [ 0.232092] acpiphp: Slot [13] registered [ 0.233129] acpiphp: Slot [14] registered [ 0.235119] acpiphp: Slot [15] registered [ 0.236101] acpiphp: Slot [16] registered [ 0.238103] acpiphp: Slot [17] registered [ 0.239135] acpiphp: Slot [18] registered [ 0.242126] acpiphp: Slot [19] registered [ 0.246154] acpiphp: Slot [20] registered [ 0.250151] acpiphp: Slot [21] registered [ 0.253144] acpiphp: Slot [22] registered [ 0.257146] acpiphp: Slot [23] registered [ 0.261163] acpiphp: Slot [24] registered [ 0.263124] acpiphp: Slot [25] registered [ 0.269155] acpiphp: Slot [26] registered [ 0.270102] acpiphp: Slot [27] registered [ 0.274148] acpiphp: Slot [28] registered [ 0.277142] acpiphp: Slot [29] registered [ 0.278000] acpiphp: Slot [30] registered [ 0.278000] acpiphp: Slot [31] registered [ 0.278087] PCI host bridge to bus 0000:00 [ 0.281000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.285032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.287018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.292000] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.292030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.294238] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.298638] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.303873] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.313016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.318617] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.324023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.330078] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334114] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.337777] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.344018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.358025] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.364019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.369968] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.384024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.416023] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.434731] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.446025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.453020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.468018] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.480599] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.487019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.495020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.517018] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.527315] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.534019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.547018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.578016] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.586214] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.592015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.597018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.610016] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.616971] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.623014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.628015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.645017] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.657000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.659384] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.660260] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.662325] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.664177] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.667305] iommu: Default domain type: Passthrough [ 0.670404] SCSI subsystem initialized [ 0.671000] ACPI: bus type USB registered [ 0.674241] usbcore: registered new interface driver usbfs [ 0.675078] usbcore: registered new interface driver hub [ 0.677078] usbcore: registered new device driver usb [ 0.679199] pps_core: LinuxPPS API ver. 1 registered [ 0.681011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.684086] PTP clock support registered [ 0.686135] EDAC MC: Ver: 3.0.0 [ 0.688142] PCI: Using ACPI for IRQ routing [ 0.689447] NetLabel: Initializing [ 0.690009] NetLabel: domain hash size = 128 [ 0.691010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.693077] NetLabel: unlabeled traffic allowed by default [ 0.695152] vgaarb: loaded [ 0.697269] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.700015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.709000] clocksource: Switched to clocksource kvm-clock [ 0.827202] VFS: Disk quotas dquot_6.6.0 [ 0.828503] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.830528] *** VALIDATE ramfs *** [ 0.831559] *** VALIDATE hugetlbfs *** [ 0.833407] pnp: PnP ACPI init [ 0.835535] pnp: PnP ACPI: found 6 devices [ 0.855250] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.858757] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.861293] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.863403] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.865682] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.867585] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.870638] NET: Registered protocol family 2 [ 0.873683] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.878522] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.883207] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.902291] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.910785] TCP: Hash tables configured (established 65536 bind 65536) [ 0.914834] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.917838] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920807] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923393] NET: Registered protocol family 1 [ 0.930241] RPC: Registered named UNIX socket transport module. [ 0.932219] RPC: Registered udp transport module. [ 0.933719] RPC: Registered tcp transport module. [ 0.935303] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.937156] NET: Registered protocol family 44 [ 0.938418] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.941276] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.943814] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.946743] PCI: CLS 0 bytes, default 64 [ 0.948949] Unpacking initramfs... [ 2.821937] debug: unmapping init [mem 0xffff8c393cc54000-0xffff8c393ffbffff] [ 2.827562] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.831301] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.838034] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.584392] Initialise system trusted keyrings [ 3.590291] Key type blacklist registered [ 3.597338] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.617079] zbud: loaded [ 3.629306] *** VALIDATE nfs *** [ 3.639366] *** VALIDATE nfs4 *** [ 3.641336] pstore: using deflate compression [ 3.659124] Platform Keyring initialized [ 3.864060] NET: Registered protocol family 38 [ 3.866719] Key type asymmetric registered [ 3.868729] Asymmetric key parser 'x509' registered [ 3.874266] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.883374] io scheduler mq-deadline registered [ 3.886233] io scheduler kyber registered [ 3.888304] io scheduler bfq registered [ 3.890400] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.895858] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.899614] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.902634] ACPI: Power Button [PWRF] [ 4.023121] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.147636] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.384975] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.523106] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.913200] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.980077] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.060241] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.069390] Non-volatile memory driver v1.3 [ 5.070835] Linux agpgart interface v0.103 [ 5.124108] virtio_blk virtio1: [vda] 133600 512-byte logical blocks (68.4 MB/65.2 MiB) [ 5.128273] vda: detected capacity change from 0 to 68403200 [ 5.172656] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.183396] vdb: detected capacity change from 0 to 1073741824 [ 5.221337] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.225986] vdc: detected capacity change from 0 to 2621440000 [ 5.261870] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.268935] vdd: detected capacity change from 0 to 2621440000 [ 5.326354] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.335035] vde: detected capacity change from 0 to 4294967296 [ 5.370188] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.373698] vdf: detected capacity change from 0 to 4294967296 [ 5.384971] libphy: Fixed MDIO Bus: probed [ 5.403086] usbcore: registered new interface driver usbserial_generic [ 5.405877] usbserial: USB Serial support registered for generic [ 5.409438] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.417138] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.419066] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.421443] mousedev: PS/2 mouse device common for all mice [ 5.424981] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.427764] rtc_cmos 00:05: RTC can wake from S4 [ 5.439417] rtc_cmos 00:05: registered as rtc0 [ 5.441386] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.445195] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.445588] intel_pstate: CPU model not supported [ 5.457286] hid: raw HID events driver (C) Jiri Kosina [ 5.461206] usbcore: registered new interface driver usbhid [ 5.464468] usbhid: USB HID core driver [ 5.465852] drop_monitor: Initializing network drop monitor service [ 5.466669] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.472186] Initializing XFRM netlink socket [ 5.472569] NET: Registered protocol family 10 [ 5.475938] Segment Routing with IPv6 [ 5.481614] NET: Registered protocol family 17 [ 5.485295] mpls_gso: MPLS GSO support [ 5.494766] RAS: Correctable Errors collector initialized. [ 5.496880] AVX version of gcm_enc/dec engaged. [ 5.499288] AES CTR mode by8 optimization enabled [ 5.636765] sched_clock: Marking stable (5636749341, 0)->(6868443211, -1231693870) [ 5.641434] registered taskstats version 1 [ 5.644178] Loading compiled-in X.509 certificates [ 5.645714] zswap: loaded using pool lzo/zbud [ 5.694889] Key type big_key registered [ 5.733177] Key type encrypted registered [ 5.734567] ima: No TPM chip found, activating TPM-bypass! [ 5.736069] ima: Allocated hash algorithm: sha1 [ 5.737698] ima: No architecture policies found [ 5.739554] evm: Initialising EVM extended attributes: [ 5.741362] evm: security.selinux [ 5.742782] evm: security.ima [ 5.743796] evm: security.capability [ 5.745264] evm: HMAC attrs: 0x1 [ 5.753757] rtc_cmos 00:05: setting system clock to 2025-11-03 10:39:46 UTC (1762166386) [ 5.760975] debug: unmapping init [mem 0xffffffff93c03000-0xffffffff93dfffff] [ 5.764364] debug: unmapping init [mem 0xffffffff92982000-0xffffffff92c58fff] [ 5.777074] Write protecting the kernel read-only data: 28672k [ 5.780590] debug: unmapping init [mem 0xffffffff91003000-0xffffffff911fffff] [ 5.783453] debug: unmapping init [mem 0xffffffff91914000-0xffffffff919fffff] [ 5.833750] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.851563] systemd[1]: Detected virtualization kvm. [ 5.855552] systemd[1]: Detected architecture x86-64. [ 5.860745] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.894566] systemd[1]: No hostname configured. [ 5.896802] systemd[1]: Set hostname to . [ 5.898692] random: systemd: uninitialized urandom read (16 bytes read) [ 5.900958] systemd[1]: Initializing machine ID from random generator. [ 6.268755] random: systemd: uninitialized urandom read (16 bytes read) [ 6.271535] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.279190] random: systemd: uninitialized urandom read (16 bytes read) [ 6.281694] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.290599] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.833721] device-mapper: uevent: version 1.0.3 [ 7.839259] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 9.539703] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.824254] random: fast init done [ 10.651084] scsi host0: ata_piix [ 10.746833] scsi host1: ata_piix [ 10.752341] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.762681] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.279603] random: crng init done [ 15.280826] random: 7 urandom warning(s) missed due to ratelimiting [ 18.021175] dracut-initqueue[600]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.248869] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.606209] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.980972] SELinux: Disabled at runtime. [ 22.049436] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.059104] systemd[1]: Detected virtualization kvm. [ 22.062233] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.022407] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.025818] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.032261] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.038325] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.041317] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.058151] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.091712] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ 23.562512] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.680728] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.086737] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.099524] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.766284] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.889905] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 30.689175] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 31.257495] NFS: Registering the id_resolver key type [ 31.259651] Key type id_resolver registered [ 31.264431] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg320-server login: [ 63.804089] libcfs: loading out-of-tree module taints kernel. [ 63.820559] Key type ._llcrypt registered [ 63.821942] Key type .llcrypt registered [ 63.877788] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 72.114465] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 72.721059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 72.727686] alg: No test for adler32 (adler32-zlib) [ 73.765189] Lustre: Lustre: Build Version: 2.16.59_50_g4204c43 [ 74.103549] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 75.719175] Key type lgssc registered [ 76.369163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.733944] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 84.194644] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.649399] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 90.862898] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.364051] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.273278] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 106.795303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.827156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 106.837265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 107.957479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 107.972713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 108.016272] Lustre: lustre-MDT0000: new disk, initializing [ 108.061619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 108.074402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 109.799889] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 115.953465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 116.002302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 116.041213] Lustre: 6495:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 116.063723] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 116.066816] Lustre: Skipped 1 previous similar message [ 116.113618] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 116.123123] Lustre: lustre-MDT0001: new disk, initializing [ 116.168302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 116.186167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 116.191039] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 117.754665] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 120.354433] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 124.372314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.408400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 124.518595] Lustre: lustre-OST0000: new disk, initializing [ 124.521884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 124.566993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 126.978690] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 129.534354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 129.538642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 129.568257] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 133.055138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 133.084459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 133.119472] Lustre: lustre-OST0001: new disk, initializing [ 133.122186] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 133.138667] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 135.157565] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 136.750873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 136.754819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 136.774507] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 141.421314] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 148.529636] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 155.905378] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing check_logdir /tmp/testlogs/ [ 157.842474] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing yml_node [ 160.088777] Lustre: DEBUG MARKER: Client: 2.16.59.50 [ 161.332570] Lustre: DEBUG MARKER: MDS: 2.16.59.50 [ 162.474268] Lustre: DEBUG MARKER: OSS: 2.16.59.50 [ 163.290972] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Mon Nov 3 05:42:23 EST 2025 [ 170.976444] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 171.831658] Lustre: DEBUG MARKER: === replay-single: start setup 05:42:32 (1762166552) === [ 173.393165] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing check_config_client /mnt/lustre [ 180.963315] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 182.302831] Lustre: 13169:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 183.703376] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.907221] Lustre: DEBUG MARKER: === replay-single: finish setup 05:42:46 (1762166566) === [ 186.625590] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 05:42:47 (1762166567) [ 189.902457] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 190.696698] Lustre: Failing over lustre-MDT0000 [ 190.768816] LustreError: 14039:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 190.804727] Lustre: server umount lustre-MDT0000 complete [ 192.992791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 192.993208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 192.997033] Lustre: Skipped 1 previous similar message [ 198.112375] LustreError: 6501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 198.119735] LustreError: 6501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 200.391227] LustreError: 6501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 203.232117] LustreError: 7992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 203.239303] LustreError: 7992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 205.510554] LustreError: 6500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 205.770756] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 205.772603] LDISKFS-fs (dm-0): recovery complete [ 205.781572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.798361] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 205.885244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.542581] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 210.637483] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 210.914644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 210.939518] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 212.444868] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 213.103555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 217.214376] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 05:43:17 (1762166597) [ 218.136080] Lustre: Failing over lustre-OST0000 [ 218.166557] LustreError: 15466:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 218.172503] LustreError: 15466:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 218.201255] Lustre: server umount lustre-OST0000 complete [ 220.886234] LustreError: 8388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 221.152857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 221.152980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 221.166300] Lustre: Skipped 3 previous similar messages [ 231.114226] LustreError: 8959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 231.126920] LustreError: 8959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 232.399833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 232.515624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.145876] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 234.485594] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 234.485942] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 234.493709] Lustre: Skipped 3 previous similar messages [ 235.186266] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.160676] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 239.856505] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 244.450740] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 05:43:44 (1762166624) [ 247.412357] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 248.344211] Lustre: Failing over lustre-MDT0000 [ 248.408141] LustreError: 17169:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 248.410180] LustreError: 17169:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 248.438656] Lustre: server umount lustre-MDT0000 complete [ 251.871835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 251.875066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 251.882624] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 251.889241] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 264.187422] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 264.189903] LDISKFS-fs (dm-0): recovery complete [ 264.195962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 264.265852] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 264.373444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 266.014434] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 267.003429] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 267.007694] Lustre: lustre-MDT0000: Denying connection for new client fdd41193-6657-4d3f-93d9-a2e32c6257fa (at 192.168.203.20@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 269.795615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 269.798103] Lustre: Skipped 1 previous similar message [ 272.070801] Lustre: lustre-MDT0000: Denying connection for new client fdd41193-6657-4d3f-93d9-a2e32c6257fa (at 192.168.203.20@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 277.190807] Lustre: lustre-MDT0000: Denying connection for new client fdd41193-6657-4d3f-93d9-a2e32c6257fa (at 192.168.203.20@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 282.312805] Lustre: lustre-MDT0000: Denying connection for new client fdd41193-6657-4d3f-93d9-a2e32c6257fa (at 192.168.203.20@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 287.431846] Lustre: lustre-MDT0000: Denying connection for new client fdd41193-6657-4d3f-93d9-a2e32c6257fa (at 192.168.203.20@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 297.672274] Lustre: lustre-MDT0000: Denying connection for new client fdd41193-6657-4d3f-93d9-a2e32c6257fa (at 192.168.203.20@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 297.681865] Lustre: Skipped 1 previous similar message [ 318.157061] Lustre: lustre-MDT0000: Denying connection for new client fdd41193-6657-4d3f-93d9-a2e32c6257fa (at 192.168.203.20@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 318.163625] Lustre: Skipped 3 previous similar messages [ 337.000272] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 337.004721] Lustre: 17820:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a6599bee-0522-4ac7-8cf6-b38305ba72e0@ [ 337.010985] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 337.021341] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 337.022444] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 337.027996] Lustre: Skipped 2 previous similar messages [ 337.047632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 337.047673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 342.670159] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 05:45:23 (1762166723) [ 345.419947] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 346.307239] Lustre: Failing over lustre-MDT0000 [ 346.372584] LustreError: 18879:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 346.376363] LustreError: 18879:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 346.412768] Lustre: server umount lustre-MDT0000 complete [ 346.592866] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.593486] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.598243] Lustre: Skipped 6 previous similar messages [ 346.608443] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 361.780637] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 361.782672] LDISKFS-fs (dm-0): recovery complete [ 361.787885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.842061] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 361.938654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 363.593858] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 364.581711] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 364.586521] Lustre: lustre-MDT0000: Denying connection for new client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 364.596953] Lustre: Skipped 3 previous similar messages [ 367.077467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 431.304664] Lustre: lustre-MDT0000: Denying connection for new client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:02 [ 431.312377] Lustre: Skipped 12 previous similar messages [ 434.000482] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 434.003081] Lustre: 19529:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client fdd41193-6657-4d3f-93d9-a2e32c6257fa@ [ 434.008140] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 434.014225] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 434.014390] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 434.017220] Lustre: Skipped 2 previous similar messages [ 434.035340] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 434.035344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 440.678598] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 05:47:01 (1762166821) [ 443.955408] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 444.721795] Lustre: Failing over lustre-MDT0000 [ 444.790797] LustreError: 20587:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 444.793246] LustreError: 20587:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 444.828034] Lustre: server umount lustre-MDT0000 complete [ 446.671380] LustreError: 6502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.676958] LustreError: 6502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 448.992220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.996188] Lustre: Skipped 3 previous similar messages [ 459.424678] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 459.426434] LDISKFS-fs (dm-0): recovery complete [ 459.431558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.485882] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 459.566385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 460.935694] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 462.022661] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 464.870696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 464.880935] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 464.899099] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 464.899109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 466.143392] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 466.765455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 470.456630] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 05:47:31 (1762166851) [ 473.118772] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 473.799731] Lustre: Failing over lustre-MDT0000 [ 473.927655] LustreError: 22408:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 473.930056] LustreError: 22408:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 473.952665] Lustre: server umount lustre-MDT0000 complete [ 475.103943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.104207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 475.109778] Lustre: Skipped 3 previous similar messages [ 488.269016] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 488.271193] LDISKFS-fs (dm-0): recovery complete [ 488.275366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 488.317418] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.812203] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 492.742710] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 493.568866] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 493.587626] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 493.587646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 494.844368] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 495.424563] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 499.124587] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 05:47:59 (1762166879) [ 501.969364] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 502.700792] Lustre: Failing over lustre-MDT0000 [ 502.841733] LustreError: 24228:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 502.844863] LustreError: 24228:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 502.884357] Lustre: server umount lustre-MDT0000 complete [ 503.775634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.775741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 503.781061] Lustre: Skipped 2 previous similar messages [ 517.458591] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 517.460269] LDISKFS-fs (dm-0): recovery complete [ 517.463685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.505632] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.343309] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 519.034965] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 522.722032] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 522.724778] Lustre: Skipped 7 previous similar messages [ 522.753984] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 522.771909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 522.771933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:193) [ 524.051876] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 524.613883] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 528.207326] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 05:48:28 (1762166908) [ 531.051831] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 531.778442] Lustre: Failing over lustre-MDT0000 [ 532.010988] Lustre: server umount lustre-MDT0000 complete [ 532.960281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 547.039203] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 547.041322] LDISKFS-fs (dm-0): recovery complete [ 547.046587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.108411] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.700718] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 552.471082] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 552.473582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 553.951099] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 554.630661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 558.250126] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 05:48:58 (1762166938) [ 560.996498] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 561.704100] Lustre: Failing over lustre-MDT0000 [ 561.899136] LustreError: 27870:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 561.902149] LustreError: 27870:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 561.931799] Lustre: server umount lustre-MDT0000 complete [ 562.657373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.657712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 562.662481] Lustre: Skipped 7 previous similar messages [ 576.601807] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 576.603601] LDISKFS-fs (dm-0): recovery complete [ 576.608331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.182753] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 579.782706] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 579.785789] Lustre: Skipped 1 previous similar message [ 582.139613] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 582.142733] Lustre: Skipped 1 previous similar message [ 582.163436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 582.163475] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 583.455650] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 584.079561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 588.132385] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 05:49:28 (1762166968) [ 588.533862] Lustre: *** cfs_fail_loc=13b, val=315*** [ 588.535249] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 588.537430] LustreError: 6503:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c3884c4b100 x1847765444449664/t38654705666(0) o35->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:0/0 lens 392/456 e 0 to 0 dl 1762166980 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 592.553281] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 593.307818] Lustre: Failing over lustre-MDT0000 [ 593.525963] Lustre: server umount lustre-MDT0000 complete [ 595.145082] LustreError: 6501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.152429] LustreError: 6501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 74 previous similar messages [ 597.471979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 608.441933] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 608.443922] LDISKFS-fs (dm-0): recovery complete [ 608.451313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.509363] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.512995] LustreError: Skipped 1 previous similar message [ 608.599062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.602473] Lustre: Skipped 4 previous similar messages [ 608.618396] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 610.098509] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 613.857765] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 613.860567] Lustre: Skipped 11 previous similar messages [ 613.880555] Lustre: 6503:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c39bf299180 x1847765444449664/t38654705666(0) o35->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:25/0 lens 392/456 e 0 to 0 dl 1762167005 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 613.892235] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 613.892448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 615.339319] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 616.052787] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 619.826195] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 05:50:00 (1762167000) [ 622.728917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 623.436564] Lustre: Failing over lustre-MDT0000 [ 623.590437] Lustre: server umount lustre-MDT0000 complete [ 638.613979] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 638.615619] LDISKFS-fs (dm-0): recovery complete [ 638.621290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.788548] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 640.405132] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 644.104703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 644.104712] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 645.607984] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 646.365270] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 650.538323] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 05:50:31 (1762167031) [ 653.532787] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 653.878747] Lustre: *** cfs_fail_loc=114, val=0*** [ 654.943825] Lustre: Failing over lustre-MDT0000 [ 655.066126] LustreError: 33481:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 655.068879] LustreError: 33481:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 655.115441] Lustre: server umount lustre-MDT0000 complete [ 659.424136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.429452] Lustre: Skipped 12 previous similar messages [ 670.092674] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 670.094343] LDISKFS-fs (dm-0): recovery complete [ 670.099923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.266688] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 671.809855] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 671.942918] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 671.948121] Lustre: Skipped 2 previous similar messages [ 675.313692] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 675.317678] Lustre: Skipped 2 previous similar messages [ 675.335562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 675.335561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 676.606465] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 677.245380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 680.909801] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 05:51:01 (1762167061) [ 683.614450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 683.964061] Lustre: *** cfs_fail_loc=128, val=0*** [ 684.944470] Lustre: Failing over lustre-MDT0000 [ 685.103749] Lustre: server umount lustre-MDT0000 complete [ 685.536127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 685.539018] LustreError: Skipped 1 previous similar message [ 699.945638] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 699.947795] LDISKFS-fs (dm-0): recovery complete [ 699.953449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.994902] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 699.999565] LustreError: Skipped 2 previous similar messages [ 700.100752] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 701.540714] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 705.531920] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 705.531931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 706.849185] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 707.452453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 711.224172] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 05:51:31 (1762167091) [ 714.322056] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 715.189202] Lustre: Failing over lustre-MDT0000 [ 715.364765] Lustre: server umount lustre-MDT0000 complete [ 730.226377] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 730.227619] LDISKFS-fs (dm-0): recovery complete [ 730.231182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.356666] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 731.835278] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 735.784300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 735.784325] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 737.048216] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 737.649133] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 741.249238] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 05:52:01 (1762167121) [ 743.854925] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 744.533216] Lustre: Failing over lustre-MDT0000 [ 744.676080] Lustre: server umount lustre-MDT0000 complete [ 759.154327] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 759.156028] LDISKFS-fs (dm-0): recovery complete [ 759.159871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.296811] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 760.780678] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 764.386351] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 764.389287] Lustre: Skipped 19 previous similar messages [ 764.433046] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 764.433056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 765.793142] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 766.421409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 770.119266] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 05:52:30 (1762167150) [ 773.045962] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 775.752315] Lustre: Failing over lustre-MDT0000 [ 775.922385] Lustre: server umount lustre-MDT0000 complete [ 790.394968] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 790.397122] LDISKFS-fs (dm-0): recovery complete [ 790.401819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.521599] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 792.004533] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 796.515342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 796.515353] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 797.873491] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 798.502086] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 809.122789] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 05:53:09 (1762167189) [ 811.905405] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 812.636020] Lustre: Failing over lustre-MDT0000 [ 812.773261] LustreError: 42705:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 812.775843] LustreError: 42705:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 812.800136] Lustre: server umount lustre-MDT0000 complete [ 816.096483] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.102165] Lustre: Skipped 19 previous similar messages [ 827.036355] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 827.038340] LDISKFS-fs (dm-0): recovery complete [ 827.041365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.166779] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 828.434806] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 830.662618] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 830.665638] Lustre: Skipped 4 previous similar messages [ 832.499775] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 832.503831] Lustre: Skipped 4 previous similar messages [ 832.522148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 832.522147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 833.744743] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 834.312416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 839.675986] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 05:53:40 (1762167220) [ 842.162908] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 842.843924] Lustre: Failing over lustre-MDT0000 [ 843.047852] Lustre: server umount lustre-MDT0000 complete [ 847.839819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 847.843034] LustreError: Skipped 3 previous similar messages [ 852.959658] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 852.966378] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 127 previous similar messages [ 857.282710] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 857.284263] LDISKFS-fs (dm-0): recovery complete [ 857.288194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.325944] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 857.329967] LustreError: Skipped 4 previous similar messages [ 858.708628] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 862.720156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 862.720192] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 863.922437] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 864.439257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 867.604155] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 05:54:08 (1762167248) [ 870.039943] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 870.655103] Lustre: Failing over lustre-MDT0000 [ 870.807468] Lustre: server umount lustre-MDT0000 complete [ 884.597092] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 884.598702] LDISKFS-fs (dm-0): recovery complete [ 884.601551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.700131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 884.703928] Lustre: Skipped 8 previous similar messages [ 886.013364] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 889.854886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 889.855040] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 890.969115] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 891.597316] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 895.169281] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 05:54:35 (1762167275) [ 897.667856] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 898.309836] Lustre: Failing over lustre-MDT0000 [ 898.457538] Lustre: server umount lustre-MDT0000 complete [ 912.883078] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 912.884956] LDISKFS-fs (dm-0): recovery complete [ 912.891600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.075910] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 913.081645] Lustre: Skipped 2 previous similar messages [ 914.589979] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 918.539684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 918.539689] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 919.797309] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 920.422049] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 924.063524] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 05:55:04 (1762167304) [ 926.731391] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 927.429307] Lustre: Failing over lustre-MDT0000 [ 927.580980] Lustre: server umount lustre-MDT0000 complete [ 941.997099] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 941.998584] LDISKFS-fs (dm-0): recovery complete [ 942.003121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.558173] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 947.201039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 947.201041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 948.482521] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 949.092806] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 952.644596] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 05:55:33 (1762167333) [ 955.304773] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 955.989026] Lustre: Failing over lustre-MDT0000 [ 956.137968] Lustre: server umount lustre-MDT0000 complete [ 970.161159] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 970.162320] LDISKFS-fs (dm-0): recovery complete [ 970.164597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.465147] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 975.357699] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 975.357770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 976.524640] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 977.065099] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 980.501754] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 05:56:01 (1762167361) [ 983.068808] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 983.757123] Lustre: Failing over lustre-MDT0000 [ 983.995500] Lustre: server umount lustre-MDT0000 complete [ 998.348774] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 998.350622] LDISKFS-fs (dm-0): recovery complete [ 998.354604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.788278] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1003.531692] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1003.531717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1004.723637] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1005.306458] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1008.840113] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 05:56:29 (1762167389) [ 1011.476921] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1012.100067] Lustre: Failing over lustre-MDT0000 [ 1012.262565] Lustre: server umount lustre-MDT0000 complete [ 1026.535052] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1026.536888] LDISKFS-fs (dm-0): recovery complete [ 1026.540208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.979765] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1031.649759] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1031.653494] Lustre: Skipped 35 previous similar messages [ 1031.674314] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1031.674314] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1032.934436] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1033.530407] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1037.156684] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 05:56:57 (1762167417) [ 1039.986335] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1040.704651] Lustre: Failing over lustre-MDT0000 [ 1040.928293] Lustre: server umount lustre-MDT0000 complete [ 1055.190110] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1055.191228] LDISKFS-fs (dm-0): recovery complete [ 1055.194107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.294278] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1055.297553] Lustre: Skipped 4 previous similar messages [ 1056.512688] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1060.341687] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 1060.341701] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 1061.476036] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1062.015522] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1065.382707] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 05:57:26 (1762167446) [ 1067.873767] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1068.521093] Lustre: Failing over lustre-MDT0000 [ 1068.673338] Lustre: server umount lustre-MDT0000 complete [ 1082.850820] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1082.852041] LDISKFS-fs (dm-0): recovery complete [ 1082.854730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.331251] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1086.662949] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1086.667146] Lustre: Skipped 8 previous similar messages [ 1087.992530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:897) [ 1087.992610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 1089.191976] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1089.754923] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1093.146873] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 05:57:53 (1762167473) [ 1095.729311] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1096.381200] Lustre: Failing over lustre-MDT0000 [ 1096.501291] LustreError: 60919:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1096.504202] LustreError: 60919:0:(obd_class.h:479:obd_check_dev()) Skipped 79 previous similar messages [ 1096.525160] Lustre: server umount lustre-MDT0000 complete [ 1098.208001] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.212167] Lustre: Skipped 39 previous similar messages [ 1110.771829] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1110.773628] LDISKFS-fs (dm-0): recovery complete [ 1110.776924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.159510] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1116.141680] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1116.145901] Lustre: Skipped 9 previous similar messages [ 1116.160206] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1116.160235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1117.299771] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1117.850626] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1121.484383] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 05:58:22 (1762167502) [ 1124.069269] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1124.739960] Lustre: Failing over lustre-MDT0000 [ 1124.888312] Lustre: server umount lustre-MDT0000 complete [ 1126.367741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1126.371477] LustreError: Skipped 9 previous similar messages [ 1138.766117] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1138.767531] LDISKFS-fs (dm-0): recovery complete [ 1138.769917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.797928] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1138.802563] LustreError: Skipped 9 previous similar messages [ 1140.135377] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1144.315829] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 1144.315858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 1145.409476] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1145.913506] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1149.110613] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 05:58:49 (1762167529) [ 1151.620568] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1152.258765] Lustre: Failing over lustre-MDT0000 [ 1152.392741] Lustre: server umount lustre-MDT0000 complete [ 1166.247817] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1166.250049] LDISKFS-fs (dm-0): recovery complete [ 1166.253276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1167.490439] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1171.445225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:993) [ 1171.445258] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 1172.446111] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1172.907016] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1176.405917] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 05:59:17 (1762167557) [ 1178.921894] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1179.611925] Lustre: Failing over lustre-MDT0000 [ 1179.721080] Lustre: server umount lustre-MDT0000 complete [ 1193.908108] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1193.909540] LDISKFS-fs (dm-0): recovery complete [ 1193.912565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.363440] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1199.101096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1199.101113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1200.255192] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1200.797389] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1204.366171] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 05:59:44 (1762167584) [ 1206.943335] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1207.640948] Lustre: Failing over lustre-MDT0000 [ 1207.809758] Lustre: server umount lustre-MDT0000 complete [ 1221.798506] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1221.800381] LDISKFS-fs (dm-0): recovery complete [ 1221.803515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.083789] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1227.257895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1227.257899] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1228.322656] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1228.857351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1232.066640] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 06:00:12 (1762167612) [ 1234.477209] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1235.074772] Lustre: Failing over lustre-MDT0000 [ 1235.207439] Lustre: server umount lustre-MDT0000 complete [ 1249.187665] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1249.189525] LDISKFS-fs (dm-0): recovery complete [ 1249.192936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.645186] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1254.394585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 1254.394596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 1255.478992] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1256.009764] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1259.195752] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 06:00:39 (1762167639) [ 1260.360253] Lustre: 71496:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting f6baae09-bf4d-49b7-9a13-98110fddd491 at adminstrative request [ 1262.538726] Lustre: Failing over lustre-MDT0000 [ 1262.689043] Lustre: server umount lustre-MDT0000 complete [ 1275.476739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.936059] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1281.015900] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 1281.015919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1282.169827] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1282.728304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1284.997378] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1287.310135] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1289.288552] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 06:01:09 (1762167669) [ 1289.612889] Lustre: 73351:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting f6baae09-bf4d-49b7-9a13-98110fddd491 at adminstrative request [ 1294.061962] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 06:01:14 (1762167674) [ 1296.367190] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1296.995115] Lustre: Failing over lustre-MDT0000 [ 1297.220606] Lustre: server umount lustre-MDT0000 complete [ 1311.076483] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1311.078452] LDISKFS-fs (dm-0): recovery complete [ 1311.081721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1312.483265] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1316.373356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 1316.373901] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 1317.527953] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1318.057579] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1321.390233] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 06:01:42 (1762167702) [ 1323.699537] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1324.322175] Lustre: Failing over lustre-MDT0000 [ 1324.462610] Lustre: server umount lustre-MDT0000 complete [ 1338.239283] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1338.240815] LDISKFS-fs (dm-0): recovery complete [ 1338.243734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.347217] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1338.349291] Lustre: Skipped 9 previous similar messages [ 1339.483841] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1343.476217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1343.476230] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1344.494601] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1344.992911] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1348.094367] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 06:02:08 (1762167728) [ 1350.341259] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1350.949368] Lustre: Failing over lustre-MDT0000 [ 1351.088850] Lustre: server umount lustre-MDT0000 complete [ 1365.130168] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1365.132291] LDISKFS-fs (dm-0): recovery complete [ 1365.135958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1366.527259] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1370.620927] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1370.620936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1371.839733] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1372.432157] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1375.958299] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 06:02:36 (1762167756) [ 1378.571788] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1379.226291] Lustre: Failing over lustre-MDT0000 [ 1379.484896] Lustre: server umount lustre-MDT0000 complete [ 1380.832342] LustreError: 6500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.837619] LustreError: 6500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 265 previous similar messages [ 1393.606876] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1393.608458] LDISKFS-fs (dm-0): recovery complete [ 1393.612320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.077660] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1398.776956] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1398.776968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1400.032407] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1400.603781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1404.273497] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 06:03:04 (1762167784) [ 1406.927125] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1407.575450] Lustre: Failing over lustre-MDT0000 [ 1407.834994] Lustre: server umount lustre-MDT0000 complete [ 1422.008322] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1422.009728] LDISKFS-fs (dm-0): recovery complete [ 1422.012529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.107787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1422.111362] Lustre: Skipped 18 previous similar messages [ 1423.465593] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1427.449334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 1427.449349] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 1428.708827] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1429.323497] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1432.880228] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 06:03:33 (1762167813) [ 1435.342560] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1435.998618] Lustre: Failing over lustre-MDT0000 [ 1436.190031] Lustre: server umount lustre-MDT0000 complete [ 1450.316969] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1450.319040] LDISKFS-fs (dm-0): recovery complete [ 1450.321395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.726091] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1455.620718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1455.620723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1456.782310] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1457.357141] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1460.695691] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 06:04:01 (1762167841) [ 1463.224954] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1463.926196] Lustre: Failing over lustre-MDT0000 [ 1464.181100] Lustre: server umount lustre-MDT0000 complete [ 1478.329554] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1478.331838] LDISKFS-fs (dm-0): recovery complete [ 1478.335550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.770351] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1483.777733] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1483.777736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1484.945690] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1485.531547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1489.213435] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 06:04:29 (1762167869) [ 1491.653438] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1492.303174] Lustre: Failing over lustre-MDT0000 [ 1492.473125] Lustre: server umount lustre-MDT0000 complete [ 1506.386103] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1506.387627] LDISKFS-fs (dm-0): recovery complete [ 1506.390539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.745277] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1511.930404] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1511.930405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1513.140752] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1513.748955] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1517.444437] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 06:04:58 (1762167898) [ 1520.121971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1520.789139] Lustre: Failing over lustre-MDT0000 [ 1520.935233] Lustre: server umount lustre-MDT0000 complete [ 1534.791445] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1534.792837] LDISKFS-fs (dm-0): recovery complete [ 1534.794947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.036481] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1540.091226] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1540.091281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1541.344037] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1541.912125] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1545.233073] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 06:05:25 (1762167925) [ 1547.594537] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1548.159823] Lustre: Failing over lustre-MDT0000 [ 1548.298699] Lustre: server umount lustre-MDT0000 complete [ 1562.356259] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1562.358618] LDISKFS-fs (dm-0): recovery complete [ 1562.361602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.746556] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1567.713246] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1567.716134] Lustre: Skipped 75 previous similar messages [ 1567.738602] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1567.738602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1568.875916] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1569.416595] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1572.818401] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 06:05:53 (1762167953) [ 1575.381692] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1576.036324] Lustre: Failing over lustre-MDT0000 [ 1576.139148] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 1576.202680] Lustre: server umount lustre-MDT0000 complete [ 1590.659951] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1590.661154] LDISKFS-fs (dm-0): recovery complete [ 1590.663890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1592.062611] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1595.899510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 1595.899530] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 1597.106592] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1597.720883] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1601.149682] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 06:06:21 (1762167981) [ 1601.511764] Lustre: 93624:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting f6baae09-bf4d-49b7-9a13-98110fddd491 at adminstrative request [ 1606.286905] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 06:06:26 (1762167986) [ 1608.553471] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1609.191051] Lustre: Failing over lustre-MDT0000 [ 1609.319494] LustreError: 94260:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1609.321241] LustreError: 94260:0:(obd_class.h:479:obd_check_dev()) Skipped 143 previous similar messages [ 1609.344914] Lustre: server umount lustre-MDT0000 complete [ 1611.231725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1611.236260] Lustre: Skipped 71 previous similar messages [ 1613.318318] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1613.319423] LDISKFS-fs (dm-0): recovery complete [ 1613.321786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.426985] Lustre: lustre-MDT0000: Aborting client recovery [ 1613.428190] LustreError: 94904:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1613.430231] Lustre: 94937:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1613.433618] Lustre: 94937:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f6baae09-bf4d-49b7-9a13-98110fddd491@ [ 1613.437585] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1613.441287] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1613.445750] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1613.464933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1505) [ 1613.464950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505) [ 1614.747505] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1618.914199] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1626.310779] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 06:06:46 (1762168006) [ 1628.903715] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1629.512638] Lustre: Failing over lustre-MDT0000 [ 1629.755601] Lustre: server umount lustre-MDT0000 complete [ 1633.649839] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1633.651247] LDISKFS-fs (dm-0): recovery complete [ 1633.654397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.741478] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1633.753672] Lustre: lustre-MDT0000: Aborting client recovery [ 1633.754915] LustreError: 96748:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1633.756719] Lustre: 96781:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1633.759094] Lustre: 96781:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 1633.760818] Lustre: 96781:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1633.763580] Lustre: 96781:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1633.766032] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1633.768220] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 1633.773072] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 1633.792958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1537) [ 1633.793442] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1537) [ 1634.928262] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1638.881658] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1644.150736] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1644.152395] Lustre: Skipped 1 previous similar message [ 1646.941092] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 06:07:07 (1762168027) [ 1651.180963] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1652.449746] Lustre: Failing over lustre-MDT0000 [ 1652.662853] Lustre: server umount lustre-MDT0000 complete [ 1654.239784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1654.244522] LustreError: Skipped 16 previous similar messages [ 1660.580668] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1660.582487] LDISKFS-fs (dm-0): recovery complete [ 1660.600606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.717651] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1660.722974] LustreError: Skipped 18 previous similar messages [ 1660.901471] Lustre: lustre-MDT0000: Aborting client recovery [ 1660.903229] LustreError: 98598:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1660.906315] Lustre: 98631:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1660.921166] Lustre: 98631:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 1660.928778] Lustre: 98631:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1660.935770] Lustre: 98631:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1660.943288] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1660.955503] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 1660.976334] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 1661.041094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 1661.044495] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 1663.889877] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1666.027331] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1682.681846] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 06:07:42 (1762168062) [ 1683.557386] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1683.559366] LustreError: 6500:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c39bc583800 x1847765445362944/t201863462916(0) o36->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:340/0 lens 512/456 e 0 to 0 dl 1762168075 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1686.862052] Lustre: Failing over lustre-MDT0000 [ 1687.029351] Lustre: server umount lustre-MDT0000 complete [ 1691.678749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.870233] Lustre: lustre-MDT0000: Aborting client recovery [ 1691.871966] LustreError: 99905:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1691.874448] Lustre: 99938:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1691.878064] Lustre: 99938:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 1691.880764] Lustre: 99938:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1691.884903] Lustre: 99938:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1691.888091] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1691.891888] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 1691.898792] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 1691.923089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 1691.923288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1601) [ 1693.490285] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1697.248437] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1706.251266] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1706.829091] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 06:08:07 (1762168087) [ 1709.538328] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1710.579426] Lustre: Failing over lustre-MDT0000 [ 1710.808105] Lustre: server umount lustre-MDT0000 complete [ 1715.843117] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1715.845178] LDISKFS-fs (dm-0): recovery complete [ 1715.849873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.968742] Lustre: lustre-MDT0000: Aborting client recovery [ 1715.970636] LustreError: 101850:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1715.973722] Lustre: 101883:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1715.976650] Lustre: 101883:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 1715.979016] Lustre: 101883:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1715.982484] Lustre: 101883:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1715.984810] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1715.987762] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 1715.992829] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 1716.013910] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 1716.014180] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1633) [ 1717.429222] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1721.314913] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1729.993541] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 06:08:30 (1762168110) [ 1744.237148] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1745.008611] Lustre: Failing over lustre-MDT0000 [ 1745.188718] Lustre: server umount lustre-MDT0000 complete [ 1759.858327] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1759.860269] LDISKFS-fs (dm-0): recovery complete [ 1759.865039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1761.591885] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1762.503114] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1762.507860] Lustre: Skipped 18 previous similar messages [ 1765.372180] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1765.377138] Lustre: Skipped 17 previous similar messages [ 1765.394970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 1765.395069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 1766.822083] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1767.454342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1777.020344] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 06:09:17 (1762168157) [ 1788.484885] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1792.513351] Lustre: Failing over lustre-MDT0000 [ 1792.760469] Lustre: server umount lustre-MDT0000 complete [ 1808.177120] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1808.178766] LDISKFS-fs (dm-0): recovery complete [ 1808.185201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1809.821847] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1814.711796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 1814.711889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 1816.188677] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1816.994187] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1827.779726] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 06:10:08 (1762168208) [ 1828.627127] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1829.054091] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1829.057660] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1831.792509] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 06:10:12 (1762168212) [ 1844.576262] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1849.642872] Lustre: Failing over lustre-OST0000 [ 1849.706053] Lustre: server umount lustre-OST0000 complete [ 1864.758125] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 1864.761295] LDISKFS-fs (dm-2): recovery complete [ 1864.767037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1864.849272] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1864.853022] Lustre: Skipped 26 previous similar messages [ 1867.308759] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1914.370878] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 06:11:34 (1762168294) [ 1917.839640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1918.938502] Lustre: Failing over lustre-MDT0000 [ 1919.150771] Lustre: server umount lustre-MDT0000 complete [ 1933.805666] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1933.807593] LDISKFS-fs (dm-0): recovery complete [ 1933.811756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.509479] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1939.469202] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 1939.469343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 1940.969423] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1941.654351] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1955.507191] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 06:12:16 (1762168336) [ 1955.807098] Lustre: 110949:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762168320/real 1762168320] req@ffff8c39856ab100 x1847765453522560/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1762168336 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 1955.815130] LustreError: 110949:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 1955.816021] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1956.831603] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 1957.398920] LustreError: 6502:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1962.463142] LustreError: 6502:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1962.466222] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnecting [ 1963.221495] LustreError: 7989:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1968.607230] LustreError: 7989:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1968.611528] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnecting [ 1969.425891] LustreError: 7989:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1974.752769] LustreError: 7989:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1974.756079] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnecting [ 1975.600276] LustreError: 7989:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1980.895141] LustreError: 7989:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1981.684345] LustreError: 19890:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1987.039115] LustreError: 19890:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1987.042456] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnecting [ 1987.046157] Lustre: Skipped 1 previous similar message [ 1993.957542] LustreError: 7989:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1993.960692] LustreError: 7989:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 1999.327132] LustreError: 7989:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1999.329762] LustreError: 7989:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 2005.471320] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnecting [ 2005.474878] Lustre: Skipped 2 previous similar messages [ 2012.287710] LustreError: 19890:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2012.291019] LustreError: 19890:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 2017.759121] LustreError: 19890:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2017.762167] LustreError: 19890:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 2021.100363] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 06:13:21 (1762168401) [ 2021.784336] LustreError: 6500:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2031.304221] Lustre: lustre-MDT0000: Export ffff8c3986280800 already connecting from 192.168.203.20@tcp [ 2036.422836] Lustre: lustre-MDT0000: Export ffff8c3986280800 already connecting from 192.168.203.20@tcp [ 2041.544653] Lustre: lustre-MDT0000: Export ffff8c3986280800 already connecting from 192.168.203.20@tcp [ 2046.662638] Lustre: lustre-MDT0000: Export ffff8c3986280800 already connecting from 192.168.203.20@tcp [ 2046.666407] Lustre: Skipped 1 previous similar message [ 2051.783397] Lustre: lustre-MDT0000: Export ffff8c3986280800 already connecting from 192.168.203.20@tcp [ 2061.839116] LustreError: 6500:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2061.842206] Lustre: 6500:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8c399df47480 x1847765447997184/t0(0) o38->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:0/0 lens 520/416 e 0 to 0 dl 1762168422 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2062.022436] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnecting [ 2062.026179] Lustre: Skipped 3 previous similar messages [ 2062.028051] LustreError: 7989:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2087.622320] Lustre: lustre-MDT0000: Export ffff8c3986280800 already connecting from 192.168.203.20@tcp [ 2087.625226] Lustre: Skipped 1 previous similar message [ 2102.071109] LustreError: 7989:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2102.074059] Lustre: 7989:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8c399dd93800 x1847765448001408/t0(0) o38->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:0/0 lens 520/416 e 0 to 0 dl 1762168462 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2102.982428] LustreError: 7989:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2128.582373] Lustre: lustre-MDT0000: Export ffff8c3986280800 already connecting from 192.168.203.20@tcp [ 2128.585506] Lustre: Skipped 2 previous similar messages [ 2143.023140] LustreError: 7989:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2143.026544] Lustre: 7989:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8c388b583800 x1847765448004480/t0(0) o38->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:0/0 lens 520/416 e 0 to 0 dl 1762168503 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2143.942233] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnecting [ 2143.944906] Lustre: Skipped 1 previous similar message [ 2143.946232] LustreError: 113200:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2169.542434] Lustre: lustre-MDT0000: Export ffff8c3986280800 already connecting from 192.168.203.20@tcp [ 2169.546336] Lustre: Skipped 2 previous similar messages [ 2183.991130] LustreError: 113200:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2183.993842] Lustre: 113200:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8c39bc360000 x1847765448007552/t0(0) o38->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:0/0 lens 520/416 e 0 to 0 dl 1762168544 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2184.902234] LustreError: 7989:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2224.943100] LustreError: 7989:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2224.946323] Lustre: 7989:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8c388bc93b80 x1847765448010624/t0(0) o38->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:0/0 lens 520/416 e 0 to 0 dl 1762168585 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2225.862287] LustreError: 10129:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2231.687047] LustreError: 10129:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 2233.221462] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 06:16:53 (1762168613) [ 2235.788248] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2237.205061] Lustre: Failing over lustre-MDT0000 [ 2237.271894] LustreError: 115035:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2237.275650] LustreError: 115035:0:(obd_class.h:479:obd_check_dev()) Skipped 67 previous similar messages [ 2237.305936] Lustre: server umount lustre-MDT0000 complete [ 2237.407677] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2237.407969] LustreError: 19890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.411202] Lustre: Skipped 35 previous similar messages [ 2237.418187] LustreError: 19890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 194 previous similar messages [ 2241.613152] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2241.614867] LDISKFS-fs (dm-0): recovery complete [ 2241.617533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.713280] Lustre: *** cfs_fail_loc=712, val=0*** [ 2241.716018] LustreError: 42023:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff8c388bc93b80 x1847765453667968/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2241.733850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2241.736442] Lustre: Skipped 15 previous similar messages [ 2241.762422] Lustre: lustre-MDT0000: Aborting client recovery [ 2241.764531] LustreError: 115678:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2241.768589] Lustre: 115711:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2241.772027] Lustre: 115711:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 2241.775073] Lustre: 115711:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2241.780074] Lustre: 115711:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2241.783427] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2241.787990] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2241.795589] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2241.819497] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2241.819508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2243.160979] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2247.136265] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2247.147443] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2247.149867] Lustre: Skipped 46 previous similar messages [ 2254.479050] Lustre: Failing over lustre-MDT0000 [ 2254.632682] Lustre: server umount lustre-MDT0000 complete [ 2257.375750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2257.379161] LustreError: Skipped 5 previous similar messages [ 2267.647554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2267.690813] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2267.694394] LustreError: Skipped 6 previous similar messages [ 2269.179863] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2273.268563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2273.268713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2274.505488] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2275.094703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2278.803085] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 06:17:39 (1762168659) [ 2278.841717] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnecting [ 2278.844445] Lustre: Skipped 2 previous similar messages [ 2282.285613] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 06:17:42 (1762168662) [ 2282.655488] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2282.657846] LustreError: 6510:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c399dc5f480 x1847765448067200/t0(0) o700->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:184/0 lens 264/248 e 0 to 0 dl 1762168674 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2299.375425] Lustre: Failing over lustre-MDT0000 [ 2299.548861] Lustre: server umount lustre-MDT0000 complete [ 2312.480373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.927745] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2317.815632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2317.815678] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2318.948428] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2319.481865] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2323.456175] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 06:18:24 (1762168704) [ 2324.337064] Lustre: Failing over lustre-OST0000 [ 2324.401177] Lustre: server umount lustre-OST0000 complete [ 2337.249388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2339.298554] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2342.486849] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2343.050253] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2408.917300] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 06:19:49 (1762168789) [ 2411.880523] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2412.741606] Lustre: Failing over lustre-MDT0000 [ 2412.948372] Lustre: server umount lustre-MDT0000 complete [ 2427.326357] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2427.327901] LDISKFS-fs (dm-0): recovery complete [ 2427.333466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2428.899982] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2431.180466] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2431.184504] Lustre: Skipped 6 previous similar messages [ 2432.600912] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2432.604073] Lustre: Skipped 6 previous similar messages [ 2432.621589] Lustre: *** cfs_fail_loc=216, val=0*** [ 2432.621647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2432.623428] LustreError: 121484:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2433.696110] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2494.997357] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 06:21:15 (1762168875) [ 2502.877338] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 06:21:23 (1762168883) [ 2503.848258] Lustre: Failing over lustre-MDT0000 [ 2504.001965] Lustre: server umount lustre-MDT0000 complete [ 2516.749561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.873558] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2516.876691] Lustre: Skipped 8 previous similar messages [ 2518.212228] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2522.083817] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2522.085622] LustreError: 123046:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c388bb60380 x1847765448183168/t0(0) o101->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:463/0 lens 328/344 e 0 to 0 dl 1762168953 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2578.630110] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnected, waiting for 2 clients in recovery for 1:06 [ 2578.634969] Lustre: 123046:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2578.653432] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 2578.653458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 2579.883733] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2580.395414] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2583.857952] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 06:22:44 (1762168964) [ 2585.203385] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2587.860195] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2588.500168] Lustre: Failing over lustre-MDT0000 [ 2588.640623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.643254] Lustre: Skipped 3 previous similar messages [ 2588.705519] Lustre: server umount lustre-MDT0000 complete [ 2602.912672] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2602.914220] LDISKFS-fs (dm-0): recovery complete [ 2602.918331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.390213] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2608.113964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3137) [ 2608.113976] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3105) [ 2609.310061] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2609.876248] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2613.499104] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 06:23:14 (1762168994) [ 2613.889721] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2617.640344] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2618.250754] Lustre: Failing over lustre-MDT0000 [ 2618.334956] Lustre: server umount lustre-MDT0000 complete [ 2632.463601] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2632.465248] LDISKFS-fs (dm-0): recovery complete [ 2632.468714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.820683] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2637.815420] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 2637.815428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3169) [ 2638.887078] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2639.436989] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2642.831115] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 06:23:43 (1762169023) [ 2643.178191] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2646.821913] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2647.433743] Lustre: Failing over lustre-MDT0000 [ 2647.522567] Lustre: server umount lustre-MDT0000 complete [ 2661.885736] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2661.887840] LDISKFS-fs (dm-0): recovery complete [ 2661.893483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.415168] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2667.508741] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3201) [ 2667.508745] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 2671.731084] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 06:24:12 (1762169052) [ 2673.069309] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2673.070270] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2673.071179] LustreError: 29253:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c398626fc50 x1847765448233344/t257698037777(0) o35->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:574/0 lens 392/456 e 0 to 0 dl 1762169064 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2673.935810] Lustre: Failing over lustre-MDT0000 [ 2674.086157] Lustre: server umount lustre-MDT0000 complete [ 2686.893587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2688.222831] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2692.075627] Lustre: 29253:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c38849a8a80 x1847765448233344/t257698037777(0) o35->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:593/0 lens 392/456 e 0 to 0 dl 1762169083 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2692.081295] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3201) [ 2692.081298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 2693.136392] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2693.694372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2697.129454] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 06:24:37 (1762169077) [ 2697.495786] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2697.498049] LustreError: 113200:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c39bc414000 x1847765448246144/t261993005072(0) o36->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:643/0 lens 504/448 e 0 to 0 dl 1762169133 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2701.266699] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2701.924302] Lustre: Failing over lustre-MDT0000 [ 2702.021849] Lustre: server umount lustre-MDT0000 complete [ 2716.197593] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2716.198843] LDISKFS-fs (dm-0): recovery complete [ 2716.202726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2717.697397] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2721.775691] Lustre: 6501:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c39bd4ebb80 x1847765448246144/t261993005072(0) o36->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:667/0 lens 504/2880 e 0 to 0 dl 1762169157 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2721.782405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 2721.782449] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3233) [ 2722.970218] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2723.532670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2727.131877] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 06:25:07 (1762169107) [ 2727.623576] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2727.625876] LustreError: 6500:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c3884c4b800 x1847765448259328/t266287972368(0) o36->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:673/0 lens 504/448 e 0 to 0 dl 1762169163 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2729.324352] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2734.887275] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2736.366077] Lustre: Failing over lustre-MDT0000 [ 2736.582434] Lustre: server umount lustre-MDT0000 complete [ 2755.567077] Lustre: 3640:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762169120/real 1762169120] req@ffff8c39856a9500 x1847765453999616/t0(0) o400->MGC192.168.203.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1762169136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2758.279562] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2758.281988] LDISKFS-fs (dm-0): recovery complete [ 2758.299251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.547415] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2771.528484] Lustre: 29253:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c388becb480 x1847765448260096/t266287972369(0) o35->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:673/0 lens 392/456 e 0 to 0 dl 1762169163 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2771.558342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 2771.570089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3265) [ 2771.575042] Lustre: 29253:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 2782.379693] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 06:26:02 (1762169162) [ 2783.261622] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2783.264046] Lustre: Skipped 1 previous similar message [ 2783.266535] LustreError: 10129:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c39bb20ca80 x1847765448275072/t270582939664(0) o36->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:729/0 lens 504/448 e 0 to 0 dl 1762169219 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2783.288980] LustreError: 10129:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 2784.860591] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2784.862630] Lustre: Skipped 1 previous similar message [ 2789.579996] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2790.431158] Lustre: Failing over lustre-MDT0000 [ 2790.533590] Lustre: server umount lustre-MDT0000 complete [ 2806.761626] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2806.763590] LDISKFS-fs (dm-0): recovery complete [ 2806.768712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.528643] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2812.415557] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 2812.415977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3329) [ 2812.428020] Lustre: 6500:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c388bece300 x1847765448275072/t270582939664(0) o36->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:3/0 lens 504/2880 e 0 to 0 dl 1762169248 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2817.159661] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 06:26:37 (1762169197) [ 2817.560480] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2818.951569] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2818.953416] LustreError: 6503:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c39bc208380 x1847765448287872/t274877906960(0) o35->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:720/0 lens 392/456 e 0 to 0 dl 1762169210 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2823.439286] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2824.326694] Lustre: Failing over lustre-MDT0000 [ 2824.392715] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 2826.416921] Lustre: server umount lustre-MDT0000 complete [ 2837.983694] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.992590] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 188 previous similar messages [ 2842.770042] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2842.772197] LDISKFS-fs (dm-0): recovery complete [ 2842.777867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2843.027709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.031407] Lustre: Skipped 12 previous similar messages [ 2844.959131] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2848.227960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2848.230496] Lustre: Skipped 48 previous similar messages [ 2848.276783] Lustre: 6503:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c39bd62a300 x1847765448287872/t274877906960(0) o35->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:750/0 lens 392/456 e 0 to 0 dl 1762169240 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2848.277813] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 2848.278045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3361) [ 2854.200584] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 06:27:14 (1762169234) [ 2854.662752] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 2854.665855] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2854.667807] Lustre: Skipped 1 previous similar message [ 2914.515986] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnecting [ 2914.521860] Lustre: Skipped 4 previous similar messages [ 2914.527528] Lustre: 10129:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c39856abb80 x1847765448298880/t279172874255(0) o101->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:105/0 lens 664/3488 e 0 to 0 dl 1762169350 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2917.995966] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 06:28:18 (1762169298) [ 2921.967719] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2923.019060] Lustre: Failing over lustre-MDT0000 [ 2923.119863] LustreError: 138993:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2923.124497] LustreError: 138993:0:(obd_class.h:479:obd_check_dev()) Skipped 107 previous similar messages [ 2923.194923] Lustre: server umount lustre-MDT0000 complete [ 2925.024804] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2925.025965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2925.032048] Lustre: Skipped 53 previous similar messages [ 2925.036524] LustreError: Skipped 10 previous similar messages [ 2939.520862] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 2939.522508] LDISKFS-fs (dm-0): recovery complete [ 2939.528932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.603715] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2939.615450] LustreError: Skipped 11 previous similar messages [ 2941.587416] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2945.050666] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 2945.050666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3393) [ 2946.912071] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2947.694404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2963.063680] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 06:29:03 (1762169343) [ 2968.247355] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2969.414839] Lustre: Failing over lustre-MDT0000 [ 2969.631973] Lustre: server umount lustre-MDT0000 complete [ 2986.202532] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2986.203819] LDISKFS-fs (dm-0): recovery complete [ 2986.209238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.608953] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2991.623532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 2991.624127] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3425) [ 2993.490178] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2994.489508] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2997.430571] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3003.980373] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 06:29:44 (1762169384) [ 3026.614215] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3027.455275] Lustre: Failing over lustre-MDT0000 [ 3027.648981] Lustre: server umount lustre-MDT0000 complete [ 3042.829041] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3042.831166] LDISKFS-fs (dm-0): recovery complete [ 3042.838325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.371169] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3044.375103] Lustre: Skipped 11 previous similar messages [ 3044.563461] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3048.500133] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3048.503759] Lustre: Skipped 11 previous similar messages [ 3048.519572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 3048.519598] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 3049.950665] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3050.601626] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3077.044760] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 06:30:57 (1762169457) [ 3081.527426] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3082.402727] Lustre: Failing over lustre-MDT0000 [ 3082.580905] Lustre: server umount lustre-MDT0000 complete [ 3099.802065] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3099.804521] LDISKFS-fs (dm-0): recovery complete [ 3099.811308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.981300] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3105.317120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 3105.317214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 3107.390709] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3108.328110] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3112.801540] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3113.726471] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3117.566444] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 06:31:37 (1762169497) [ 3124.087260] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3185.883280] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3185.885558] LustreError: 19890:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c399e0f0000 x1847765451089792/t296352743435(0) o36->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:376/0 lens 66040/440 e 0 to 0 dl 1762169621 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3185.895016] LustreError: 19890:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 3245.258678] Lustre: 19890:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c398360bc50 x1847765451089792/t296352743435(0) o36->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:436/0 lens 66040/440 e 0 to 0 dl 1762169681 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3249.053325] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3249.688955] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 06:33:50 (1762169630) [ 3254.889608] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3255.969846] Lustre: Failing over lustre-MDT0000 [ 3256.221399] Lustre: server umount lustre-MDT0000 complete [ 3271.470567] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3271.472196] LDISKFS-fs (dm-0): recovery complete [ 3271.481308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.717074] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3271.721722] Lustre: Skipped 12 previous similar messages [ 3273.462328] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3277.132565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 3277.132610] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 3278.595865] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3279.277624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3283.644932] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 06:34:24 (1762169664) [ 3292.242105] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3297.232505] Lustre: Failing over lustre-OST0000 [ 3297.249340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3297.252087] Lustre: Skipped 1 previous similar message [ 3297.277992] Lustre: server umount lustre-OST0000 complete [ 3313.141843] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3313.143722] LDISKFS-fs (dm-2): recovery complete [ 3313.149145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3315.693078] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3327.723964] Lustre: Failing over lustre-OST0000 [ 3327.782759] Lustre: server umount lustre-OST0000 complete [ 3341.683322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3344.326336] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3348.310318] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3349.095282] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3384.219554] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 06:36:04 (1762169764) [ 3385.395623] Lustre: Failing over lustre-MDT0000 [ 3385.637909] Lustre: server umount lustre-MDT0000 complete [ 3386.849276] LustreError: 8404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762169767 with bad export cookie 16875287510331380384 [ 3399.353215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3401.195333] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3404.793667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 3404.793669] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 3413.183441] Lustre: Failing over lustre-MDT0000 [ 3413.406219] Lustre: server umount lustre-MDT0000 complete [ 3427.426496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.626326] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3432.957245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 3432.957990] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 3434.754632] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3435.666648] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3440.101944] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 06:37:00 (1762169820) [ 3451.977184] Lustre: Failing over lustre-OST0000 [ 3452.017303] Lustre: server umount lustre-OST0000 complete [ 3453.408231] LustreError: 16727:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3453.415171] LustreError: 16727:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 139 previous similar messages [ 3465.761759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3465.857713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3465.861467] Lustre: Skipped 9 previous similar messages [ 3467.704830] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3467.707214] Lustre: Skipped 35 previous similar messages [ 3468.233503] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3472.023853] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3472.844332] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3477.351867] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 06:37:37 (1762169857) [ 3478.154830] Lustre: Failing over lustre-MDT0000 [ 3478.402339] Lustre: server umount lustre-MDT0000 complete [ 3482.480067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.535746] Lustre: *** cfs_fail_loc=605, val=0*** [ 3482.537104] LustreError: 156822:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0883860 failed: rc = -95 [ 3482.540819] LustreError: 156822:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 3482.543367] LustreError: 156822:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 3482.547021] LustreError: 156822:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 3482.550169] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3482.553338] LustreError: 156822:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3482.558879] Lustre: server umount lustre-MDT0000 complete [ 3482.560422] LustreError: 156822:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3485.879867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.684543] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3491.049833] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 06:37:51 (1762169871) [ 3491.325213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3491.325403] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3494.675749] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3496.118598] Lustre: Failing over lustre-MDT0000 [ 3496.136656] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 3496.338285] Lustre: server umount lustre-MDT0000 complete [ 3511.820859] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3511.822859] LDISKFS-fs (dm-0): recovery complete [ 3511.827954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.695090] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3516.620630] Lustre: *** cfs_fail_loc=707, val=0*** [ 3576.020863] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 3576.037977] Lustre: 158976:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3576.042444] Lustre: 158976:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 3576.179590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 3576.179643] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 3577.865444] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3578.527315] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3583.267393] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 06:39:23 (1762169963) [ 3607.949630] LustreError: 10129:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8c39bd12b480 x1847765451971584/t0(0) o101->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:754/0 lens 664/0 e 0 to 0 dl 1762169999 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3607.959867] LustreError: 10129:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3618.991146] LustreError: 10129:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3618.998749] LustreError: 29253:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8c39812ab100 x1847765451972864/t0(0) o35->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:10/0 lens 392/0 e 0 to 0 dl 1762170010 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3624.929949] LustreError: 16727:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8c388ca36680 x1847765455423104/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:16/0 lens 544/0 e 0 to 0 dl 1762170016 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 3624.939145] LustreError: 16727:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 3631.380581] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 06:40:11 (1762170011) [ 3655.033132] LustreError: 8392:0:(tgt_handler.c:2784:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3666.063732] LustreError: 8392:0:(tgt_handler.c:2784:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3669.301319] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 06:40:49 (1762170049) [ 3692.498276] LustreError: 6500:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8c399e0f2d80 x1847765451996800/t0(0) o101->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:84/0 lens 576/0 e 0 to 0 dl 1762170084 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3692.507112] LustreError: 6500:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3697.607142] LustreError: 6500:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3697.613651] LustreError: 6500:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8c399e21b480 x1847765451997696/t0(0) o101->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:89/0 lens 664/0 e 0 to 0 dl 1762170089 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3708.639109] LustreError: 6501:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3708.644989] LustreError: 6501:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8c399dd8c380 x1847765452019456/t0(0) o101->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:139/0 lens 664/0 e 0 to 0 dl 1762170139 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3708.654830] LustreError: 6501:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 3721.370549] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 06:41:41 (1762170101) [ 3807.425181] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 06:43:07 (1762170187) [ 3830.883810] LustreError: 6500:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8c39bd5a5f80 x1847765452076672/t0(0) o101->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:222/0 lens 576/0 e 0 to 0 dl 1762170222 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3830.892952] LustreError: 6500:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 3830.897114] LustreError: 6500:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3830.900360] LustreError: 6500:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 3831.319152] LustreError: 6500:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3847.144827] LustreError: 6501:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3847.148324] LustreError: 6501:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3847.567067] LustreError: 6501:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3847.570778] LustreError: 6501:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3862.977238] LustreError: 19890:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff8c39b77c8380 x1847765452095872/t0(0) o36->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:254/0 lens 504/0 e 0 to 0 dl 1762170254 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 3862.986036] LustreError: 19890:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 82 previous similar messages [ 3875.289772] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 06:44:15 (1762170255) [ 3899.973923] Lustre: DEBUG MARKER: phase 2 [ 3903.094781] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 06:44:43 (1762170283) [ 3973.797199] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 06:45:54 (1762170354) [ 3974.277662] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 3974.807795] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 06:45:55 (1762170355) [ 3976.512815] Lustre: DEBUG MARKER: Started rundbench load pid=133564 ... [ 3980.109203] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3981.596164] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 3982.249891] Lustre: Failing over lustre-MDT0000 [ 3982.432713] LustreError: 165872:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3982.435952] LustreError: 165872:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages [ 3982.467705] Lustre: server umount lustre-MDT0000 complete [ 3983.327711] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3983.330675] Lustre: Skipped 43 previous similar messages [ 3996.762775] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3996.764184] LDISKFS-fs (dm-0): recovery complete [ 3996.767539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.801025] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3996.803622] LustreError: Skipped 8 previous similar messages [ 3996.869829] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3996.871983] Lustre: Skipped 7 previous similar messages [ 3997.893956] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3997.896219] Lustre: Skipped 9 previous similar messages [ 3998.102242] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4002.284259] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4002.286307] Lustre: Skipped 9 previous similar messages [ 4002.300262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 4002.300264] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 4003.478232] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4004.081713] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4008.897416] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4010.404620] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 4011.044714] Lustre: Failing over lustre-MDT0001 [ 4011.053676] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 4011.252738] Lustre: server umount lustre-MDT0001 complete [ 4012.512581] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 4012.515037] LustreError: Skipped 8 previous similar messages [ 4025.187082] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 4025.188601] LDISKFS-fs (dm-1): recovery complete [ 4025.193413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4026.671190] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4030.791305] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:827 to 0x280000400:865) [ 4030.791315] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:809 to 0x2c0000400:833) [ 4032.096833] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4032.699448] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4037.747835] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4039.250965] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 4039.879967] Lustre: Failing over lustre-MDT0000 [ 4040.123787] Lustre: server umount lustre-MDT0000 complete [ 4054.361060] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4054.362421] LDISKFS-fs (dm-0): recovery complete [ 4054.366727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4055.931467] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4059.642675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 4059.642816] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 4060.874405] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4061.453811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4066.208974] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4067.744276] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 4068.417658] Lustre: Failing over lustre-MDT0001 [ 4068.423873] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 4068.577047] Lustre: server umount lustre-MDT0001 complete [ 4069.856346] LustreError: 19890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.862308] LustreError: 19890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 77 previous similar messages [ 4082.576252] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 4082.577877] LDISKFS-fs (dm-1): recovery complete [ 4082.581635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4082.665534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4082.667971] Lustre: Skipped 5 previous similar messages [ 4083.839315] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4087.776947] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4087.779937] Lustre: Skipped 20 previous similar messages [ 4088.065686] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1270 to 0x280000400:1313) [ 4088.065709] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1239 to 0x2c0000400:1281) [ 4089.288196] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4089.872845] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4094.733462] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4096.259639] Lustre: DEBUG MARKER: test_70b fail mds1 5 times [ 4096.953271] Lustre: Failing over lustre-MDT0000 [ 4097.138589] Lustre: server umount lustre-MDT0000 complete [ 4111.198278] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4111.199708] LDISKFS-fs (dm-0): recovery complete [ 4111.203194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4112.614929] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4116.477096] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5121) [ 4116.477156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5153) [ 4117.616757] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4118.156060] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4121.903376] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 06:48:22 (1762170502) [ 4244.887701] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4255.542313] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 4256.127306] Lustre: Failing over lustre-MDT0000 [ 4256.150096] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 4256.324172] Lustre: server umount lustre-MDT0000 complete [ 4270.171758] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4270.173455] LDISKFS-fs (dm-0): recovery complete [ 4270.178868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.566450] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4278.115443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6589 to 0x280000401:6625) [ 4278.115764] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6558 to 0x2c0000401:6593) [ 4279.316241] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4279.861489] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4404.092505] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4414.876823] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 4415.544271] Lustre: Failing over lustre-MDT0001 [ 4415.554512] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 4421.201200] Lustre: server umount lustre-MDT0001 complete [ 4435.254326] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4435.255411] LDISKFS-fs (dm-1): recovery complete [ 4435.258657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4436.572038] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4442.541479] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4320 to 0x280000400:4353) [ 4442.547289] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4288 to 0x2c0000400:4321) [ 4443.834089] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4444.368415] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4456.464291] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 06:53:57 (1762170837) [ 4579.282198] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4590.032364] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 4590.655971] Lustre: Failing over lustre-MDT0000 [ 4590.666259] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 4590.669488] Lustre: Skipped 4 previous similar messages [ 4590.885404] LustreError: 178163:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4590.887778] LustreError: 178163:0:(obd_class.h:479:obd_check_dev()) Skipped 55 previous similar messages [ 4590.948401] Lustre: server umount lustre-MDT0000 complete [ 4593.123406] LustreError: 42030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762170973 with bad export cookie 16875287510335056924 [ 4593.124201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4593.126980] LustreError: 42030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4593.130640] Lustre: Skipped 21 previous similar messages [ 4605.374146] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4605.376318] LDISKFS-fs (dm-0): recovery complete [ 4605.380715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4605.425908] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4605.429040] LustreError: Skipped 3 previous similar messages [ 4605.515582] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4605.521717] Lustre: Skipped 6 previous similar messages [ 4606.957231] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4610.246177] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4610.249920] Lustre: Skipped 6 previous similar messages [ 4611.690350] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4611.693234] Lustre: Skipped 6 previous similar messages [ 4611.716530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8464 to 0x280000401:8481) [ 4611.716565] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8432 to 0x2c0000401:8449) [ 4612.959662] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4613.551756] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4737.792595] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4748.605098] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 4749.298602] Lustre: Failing over lustre-MDT0000 [ 4749.308145] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 4749.622577] Lustre: server umount lustre-MDT0000 complete [ 4750.304421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4750.307454] LustreError: Skipped 3 previous similar messages [ 4750.309651] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4750.315358] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 65 previous similar messages [ 4763.569426] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4763.571410] LDISKFS-fs (dm-0): recovery complete [ 4763.575226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4763.672479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4763.674232] Lustre: Skipped 4 previous similar messages [ 4764.847023] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4768.737637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4768.739274] Lustre: Skipped 17 previous similar messages [ 4769.716698] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8893 to 0x2c0000401:8929) [ 4769.717234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8926 to 0x280000401:8961) [ 4770.903539] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4771.465498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4774.987259] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 06:59:15 (1762171155) [ 4898.112809] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4908.919203] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 4909.596092] Lustre: Failing over lustre-MDT0000 [ 4909.808349] Lustre: server umount lustre-MDT0000 complete [ 4924.155756] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4924.157648] LDISKFS-fs (dm-0): recovery complete [ 4924.160960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4925.613275] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4932.944358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10862 to 0x280000401:10881) [ 4932.946179] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10829 to 0x2c0000401:10849) [ 4934.148025] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4934.705682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5058.655150] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5069.343619] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 5069.929667] Lustre: Failing over lustre-MDT0000 [ 5070.239428] Lustre: server umount lustre-MDT0000 complete [ 5083.935073] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5083.936154] LDISKFS-fs (dm-0): recovery complete [ 5083.938708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5085.307562] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5092.701500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12910 to 0x280000401:12929) [ 5092.701578] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12877 to 0x2c0000401:12897) [ 5094.015788] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5094.656317] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5098.136601] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 07:04:38 (1762171478) [ 5103.674291] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5105.109656] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5105.664222] Lustre: Failing over lustre-OST0000 [ 5105.705547] Lustre: server umount lustre-OST0000 complete [ 5119.825959] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5119.827765] LDISKFS-fs (dm-2): recovery complete [ 5119.831091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5121.798850] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5125.141410] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5125.843428] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5134.768195] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5136.294940] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5136.945759] Lustre: Failing over lustre-OST0000 [ 5136.987323] Lustre: server umount lustre-OST0000 complete [ 5150.849973] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5150.851196] LDISKFS-fs (dm-2): recovery complete [ 5150.854375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5152.712391] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5155.939630] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5156.557102] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5162.265773] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 07:05:42 (1762171542) [ 5285.288830] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5287.915207] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5298.619878] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 5299.263885] Lustre: Failing over lustre-MDT0001 [ 5299.265942] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5299.269769] Lustre: Skipped 19 previous similar messages [ 5299.271554] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5299.481597] LustreError: 189591:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5299.483639] LustreError: 189591:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 5299.572478] Lustre: server umount lustre-MDT0001 complete [ 5300.797364] Lustre: Failing over lustre-MDT0000 [ 5306.976311] Lustre: server umount lustre-MDT0000 complete [ 5321.766928] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5321.769118] LDISKFS-fs (dm-0): recovery complete [ 5321.773927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.819290] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5321.826766] LustreError: Skipped 3 previous similar messages [ 5321.829647] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5321.830655] LDISKFS-fs (dm-1): recovery complete [ 5321.834425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5321.947029] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5321.950804] Lustre: Skipped 5 previous similar messages [ 5323.373388] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5323.491391] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5325.152776] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5325.155141] Lustre: Skipped 6 previous similar messages [ 5331.147267] Lustre: lustre-MDT0001: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 5331.149634] Lustre: Skipped 5 previous similar messages [ 5331.166821] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5249 to 0x2c0000400:5281) [ 5331.166852] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5281 to 0x280000400:5313) [ 5341.196655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12990 to 0x280000401:13025) [ 5341.202152] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:12993) [ 5342.622913] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5343.277655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5343.859120] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5347.350246] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 07:08:47 (1762171727) [ 5349.886437] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5350.786496] Lustre: Failing over lustre-MDT0000 [ 5351.020587] Lustre: server umount lustre-MDT0000 complete [ 5351.391575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5351.394028] LustreError: Skipped 7 previous similar messages [ 5351.396662] LustreError: 173884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5351.402508] LustreError: 173884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 68 previous similar messages [ 5365.156337] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5365.157702] LDISKFS-fs (dm-0): recovery complete [ 5365.160802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.261191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5365.263116] Lustre: Skipped 6 previous similar messages [ 5366.528709] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5366.996549] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5370.337428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5370.339724] Lustre: Skipped 21 previous similar messages [ 5383.377296] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnected, waiting for 2 clients in recovery for 0:52 [ 5383.388269] Lustre: 191681:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c3891347b80 x1847765544546432/t352187318885(352187318885) o101->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:265/0 lens 520/3488 e 0 to 0 dl 1762171775 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5383.415552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13027 to 0x280000401:13057) [ 5383.415581] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13025) [ 5384.786284] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5385.331737] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5388.745525] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 07:09:29 (1762171769) [ 5391.350730] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5392.307993] Lustre: Failing over lustre-MDT0000 [ 5392.474262] Lustre: server umount lustre-MDT0000 complete [ 5406.425754] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5406.427180] LDISKFS-fs (dm-0): recovery complete [ 5406.430252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5407.771909] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5407.950375] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5407.951606] LustreError: 190907:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c38933bd880 x1847765544546432/t352187318885(352187318885) o101->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:289/0 lens 520/664 e 0 to 0 dl 1762171799 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5424.332716] Lustre: lustre-MDT0000: Client f6baae09-bf4d-49b7-9a13-98110fddd491 (at 192.168.203.20@tcp) reconnected, waiting for 2 clients in recovery for 0:52 [ 5424.336908] Lustre: 191681:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c38831bbb80 x1847765544546432/t352187318885(352187318885) o101->f6baae09-bf4d-49b7-9a13-98110fddd491@192.168.203.20@tcp:306/0 lens 520/3488 e 0 to 0 dl 1762171816 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5424.365492] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13057) [ 5424.365492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13089) [ 5425.662040] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5426.208824] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5429.500218] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 07:10:10 (1762171810) [ 5430.848744] Lustre: Failing over lustre-OST0000 [ 5430.890932] Lustre: server umount lustre-OST0000 complete [ 5432.051033] Lustre: Failing over lustre-MDT0000 [ 5432.136132] Lustre: server umount lustre-MDT0000 complete [ 5444.883739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5446.312824] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5447.035575] Lustre: lustre-MDT0000: Denying connection for new client ee6f2298-aae5-4b41-830b-45711a5740a8 (at 192.168.203.20@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5450.226735] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13089) [ 5454.343162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5456.164712] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5456.373047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13059 to 0x280000401:13121) [ 5460.914884] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 07:10:41 (1762171841) [ 5461.256692] LustreError: 8398:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c39b2107b80 x1847765501270016/t369367187463(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:343/0 lens 1312/4320 e 0 to 0 dl 1762171853 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5463.721228] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5464.312545] Lustre: Failing over lustre-MDT0000 [ 5464.593263] Lustre: server umount lustre-MDT0000 complete [ 5477.855121] Lustre: 190962:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762171842/real 1762171842] req@ffff8c39b2106d80 x1847765501270016/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1762171858 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5478.523756] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5478.524953] LDISKFS-fs (dm-0): recovery complete [ 5478.527330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5479.896407] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5484.023311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13153) [ 5484.023327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13121) [ 5485.121611] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5485.662356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5489.096099] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 07:11:09 (1762171869) [ 5489.425982] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5489.427289] Lustre: Skipped 1 previous similar message [ 5491.811614] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5494.246314] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5494.851831] Lustre: Failing over lustre-MDT0001 [ 5494.854526] LustreError: 200664:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x20002a3e1:0x4:0x0].0x0 (ffff8c3988eac000) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5501.006861] Lustre: server umount lustre-MDT0001 complete [ 5514.984559] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5514.985823] LDISKFS-fs (dm-1): recovery complete [ 5514.989718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5516.389443] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5520.366098] Lustre: 190907:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c3885917800 x1847765551705728/t21474839086(0) o36->ee6f2298-aae5-4b41-830b-45711a5740a8@192.168.203.20@tcp:424/0 lens 560/2880 e 0 to 0 dl 1762171934 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5520.372948] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5292 to 0x2c0000400:5313) [ 5520.372956] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5324 to 0x280000400:5345) [ 5521.516287] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5522.070403] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5525.630395] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 07:11:46 (1762171906) [ 5525.972769] LustreError: 173883:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c39b237d880 x1847765501330176/t373662154785(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:407/0 lens 2520/4320 e 0 to 0 dl 1762171917 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5525.985257] LustreError: 173883:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 5528.395833] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5530.801976] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5531.403071] Lustre: Failing over lustre-MDT0000 [ 5531.729351] Lustre: server umount lustre-MDT0000 complete [ 5541.343151] Lustre: 201289:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762171906/real 1762171906] req@ffff8c39b237df80 x1847765501330176/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1762171922 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5545.644107] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5545.645418] LDISKFS-fs (dm-0): recovery complete [ 5545.648675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5547.040400] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5551.127194] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13153) [ 5551.127203] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13185) [ 5552.258311] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5552.787111] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5554.675575] Lustre: Failing over lustre-MDT0001 [ 5554.765297] Lustre: server umount lustre-MDT0001 complete [ 5568.622475] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5568.623576] LDISKFS-fs (dm-1): recovery complete [ 5568.627484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5569.992847] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5574.132664] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5356 to 0x280000400:5377) [ 5574.132732] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5324 to 0x2c0000400:5345) [ 5575.272138] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5575.806865] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5579.222938] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 07:12:39 (1762171959) [ 5584.805222] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5587.112052] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5587.712790] Lustre: Failing over lustre-MDT0000 [ 5587.955973] Lustre: server umount lustre-MDT0000 complete [ 5589.089746] LustreError: 144665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762171969 with bad export cookie 16875287510340190983 [ 5589.091092] Lustre: Failing over lustre-MDT0001 [ 5589.093447] LustreError: 144665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5595.104088] LustreError: 206593:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8c388b223100 x1847765501389952/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5595.109537] LustreError: 206593:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5595.200805] Lustre: server umount lustre-MDT0001 complete [ 5609.673899] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5609.676853] LDISKFS-fs (dm-0): recovery complete [ 5609.681599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5609.682382] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5609.686832] LDISKFS-fs (dm-1): recovery complete [ 5609.690756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5614.047298] LustreError: 3639:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c38931d3100 x1847765501390720/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5615.499295] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5615.590528] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5620.453779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13217) [ 5620.453793] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13185) [ 5625.557146] Lustre: 207712:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c3885915f80 x1847765551773312/t30064771140(0) o36->ee6f2298-aae5-4b41-830b-45711a5740a8@192.168.203.20@tcp:532/0 lens 560/2880 e 0 to 0 dl 1762172042 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5625.564957] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5388 to 0x280000400:5409) [ 5625.564987] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5356 to 0x2c0000400:5377) [ 5626.682213] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5627.202928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5627.679132] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5631.096657] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 07:13:31 (1762172011) [ 5631.419066] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5631.420629] Lustre: Skipped 2 previous similar messages [ 5631.421816] LustreError: 207710:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c39bc179180 x1847765551805824/t34359738436(0) o36->ee6f2298-aae5-4b41-830b-45711a5740a8@192.168.203.20@tcp:552/0 lens 560/448 e 0 to 0 dl 1762172062 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5631.426926] LustreError: 207710:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 5636.765237] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5637.323843] Lustre: Failing over lustre-MDT0000 [ 5637.582810] Lustre: server umount lustre-MDT0000 complete [ 5651.397651] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5651.399384] LDISKFS-fs (dm-0): recovery complete [ 5651.402453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5652.686712] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5656.566340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13249) [ 5656.566372] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13217) [ 5657.634464] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5658.130980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5686.470320] Lustre: lustre-MDT0001: Client ee6f2298-aae5-4b41-830b-45711a5740a8 (at 192.168.203.20@tcp) reconnecting [ 5686.474288] Lustre: Skipped 2 previous similar messages [ 5686.477594] Lustre: 208450:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c38931d1f80 x1847765551805824/t34359738436(0) o36->ee6f2298-aae5-4b41-830b-45711a5740a8@192.168.203.20@tcp:607/0 lens 560/2880 e 0 to 0 dl 1762172117 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5688.823653] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 07:14:29 (1762172069) [ 5691.530537] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5692.144897] Lustre: Failing over lustre-MDT0001 [ 5692.227108] Lustre: server umount lustre-MDT0001 complete [ 5706.153752] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5706.155721] LDISKFS-fs (dm-1): recovery complete [ 5706.160369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5707.434453] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5711.346042] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5398 to 0x2c0000400:5441) [ 5711.346042] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5430 to 0x280000400:5473) [ 5712.438195] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5712.929610] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5716.291846] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 07:14:56 (1762172096) [ 5721.932234] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5724.234174] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5724.828886] Lustre: Failing over lustre-MDT0000 [ 5725.094508] Lustre: server umount lustre-MDT0000 complete [ 5738.857841] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5738.858829] LDISKFS-fs (dm-0): recovery complete [ 5738.861526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.134053] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5744.169877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13281) [ 5744.169921] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13249) [ 5746.007881] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5746.855561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5749.354070] Lustre: Failing over lustre-MDT0001 [ 5749.460551] Lustre: server umount lustre-MDT0001 complete [ 5768.775790] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5768.777886] LDISKFS-fs (dm-1): recovery complete [ 5768.792925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5772.657151] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5774.387957] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5484 to 0x280000400:5505) [ 5774.398893] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5452 to 0x2c0000400:5473) [ 5778.944386] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5780.547583] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5786.128641] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 07:16:06 (1762172166) [ 5786.587350] LustreError: 207712:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c388c0cca80 x1847765551905664/t42949673030(0) o36->ee6f2298-aae5-4b41-830b-45711a5740a8@192.168.203.20@tcp:668/0 lens 560/448 e 0 to 0 dl 1762172178 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5786.597950] LustreError: 207712:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 2 previous similar messages [ 5792.289977] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5795.050559] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5795.747553] Lustre: Failing over lustre-MDT0000 [ 5795.987682] Lustre: server umount lustre-MDT0000 complete [ 5797.287607] LustreError: 7442:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172178 with bad export cookie 16875287510340204640 [ 5797.288686] Lustre: Failing over lustre-MDT0001 [ 5797.292718] LustreError: 7442:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5797.398533] Lustre: server umount lustre-MDT0001 complete [ 5812.275467] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5812.278057] LDISKFS-fs (dm-0): recovery complete [ 5812.281626] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5812.282609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5812.284401] LDISKFS-fs (dm-1): recovery complete [ 5812.296419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5815.775396] Lustre: 3640:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762172180/real 1762172180] req@ffff8c39bfb58700 x1847765501558784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762172196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5815.785418] Lustre: 3640:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5818.847230] Lustre: 3643:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762172183/real 1762172183] req@ffff8c39b2104700 x1847765501559552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762172199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5818.855501] Lustre: 3643:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5822.946590] LustreError: 3639:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c39be059c00 x1847765501561600/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5824.223124] Lustre: 3640:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762172188/real 1762172188] req@ffff8c39bfb59880 x1847765501559808/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762172204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5824.232814] Lustre: 3640:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5824.728696] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5824.838199] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5827.318483] Lustre: 218312:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c39bc152680 x1847765551905664/t42949673030(0) o36->ee6f2298-aae5-4b41-830b-45711a5740a8@192.168.203.20@tcp:709/0 lens 560/2880 e 0 to 0 dl 1762172219 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5827.320573] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5484 to 0x2c0000400:5505) [ 5827.320582] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5516 to 0x280000400:5537) [ 5827.326933] Lustre: 218312:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages [ 5836.465186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13313) [ 5836.465584] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13281) [ 5837.960521] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5838.557431] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5839.087281] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5843.031759] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 07:17:03 (1762172223) [ 5846.027434] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5846.740409] Lustre: Failing over lustre-MDT0001 [ 5846.744377] LustreError: 219990:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x20002a3e1:0x17:0x0].0xf7117594 (ffff8c398850ce00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5846.755488] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 5846.759039] Lustre: Skipped 21 previous similar messages [ 5853.239308] Lustre: server umount lustre-MDT0001 complete [ 5867.932324] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5867.934132] LDISKFS-fs (dm-1): recovery complete [ 5867.940348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5870.066941] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5873.171924] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5516 to 0x2c0000400:5537) [ 5873.172182] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5548 to 0x280000400:5569) [ 5875.376766] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5876.476486] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5883.607396] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 07:17:43 (1762172263) [ 5892.308287] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5894.101715] Lustre: Failing over lustre-MDT0000 [ 5894.409905] Lustre: server umount lustre-MDT0000 complete [ 5900.255173] Lustre: 220618:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762172265/real 1762172265] req@ffff8c3885914380 x1847765501629056/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1984/4320 e 0 to 1 dl 1762172281 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5900.278192] Lustre: 220618:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 5913.654208] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5913.655745] LDISKFS-fs (dm-0): recovery complete [ 5913.663700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5915.796181] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5919.211166] Lustre: 222501:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5919.216484] Lustre: 222501:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 52 previous similar messages [ 5919.242567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13345) [ 5919.242810] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13313) [ 5920.926213] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5921.830168] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5926.540274] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 07:18:26 (1762172306) [ 5927.138445] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5927.140672] Lustre: Skipped 5 previous similar messages [ 5931.117033] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5935.109925] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5936.011171] Lustre: Failing over lustre-MDT0000 [ 5936.189237] LustreError: 224116:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 5936.192395] LustreError: 224116:0:(obd_class.h:479:obd_check_dev()) Skipped 155 previous similar messages [ 5936.225943] Lustre: server umount lustre-MDT0000 complete [ 5939.680616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5939.686566] Lustre: Skipped 62 previous similar messages [ 5943.263156] Lustre: 220618:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762172307/real 1762172307] req@ffff8c388c28d880 x1847765501658112/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1728/4320 e 0 to 1 dl 1762172323 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5953.433672] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5953.435911] LDISKFS-fs (dm-0): recovery complete [ 5953.442483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5953.511874] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5953.516655] LustreError: Skipped 10 previous similar messages [ 5953.678709] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5953.681645] Lustre: Skipped 19 previous similar messages [ 5955.991446] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5956.298794] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5956.301569] Lustre: Skipped 18 previous similar messages [ 5959.155266] Lustre: 224768:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5959.168979] Lustre: 224768:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 4 previous similar messages [ 5959.207236] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5959.217105] Lustre: Skipped 19 previous similar messages [ 5959.240318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13377) [ 5959.240820] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13345) [ 5961.081918] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5961.996268] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5964.516023] Lustre: Failing over lustre-MDT0001 [ 5964.664770] Lustre: server umount lustre-MDT0001 complete [ 5966.541095] LustreError: 218311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.550281] LustreError: 218311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 197 previous similar messages [ 5968.863592] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5968.867583] LustreError: Skipped 13 previous similar messages [ 5980.870722] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5980.872615] LDISKFS-fs (dm-1): recovery complete [ 5980.890093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5981.066201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5981.071338] Lustre: Skipped 18 previous similar messages [ 5983.124649] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5986.276995] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5986.280337] Lustre: Skipped 63 previous similar messages [ 5986.320670] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5548 to 0x280000400:5601) [ 5986.321401] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5516 to 0x2c0000400:5569) [ 5987.898813] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5988.679716] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5992.928427] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 07:19:33 (1762172373) [ 5997.046504] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6000.801595] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6001.730201] Lustre: Failing over lustre-MDT0000 [ 6001.986464] Lustre: server umount lustre-MDT0000 complete [ 6003.798146] LustreError: 6485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172384 with bad export cookie 16875287510340215049 [ 6003.799326] Lustre: Failing over lustre-MDT0001 [ 6003.802238] LustreError: 6485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6003.805452] LustreError: 227797:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x20002a3e1:0x1f:0x0].0x0 (ffff8c3883883e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6009.824265] LustreError: 227797:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8c39b244b480 x1847765501716352/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6009.831541] LustreError: 227797:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6009.961535] Lustre: server umount lustre-MDT0001 complete [ 6025.157509] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6025.159483] LDISKFS-fs (dm-0): recovery complete [ 6025.163535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.179902] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6025.182180] LDISKFS-fs (dm-1): recovery complete [ 6025.187226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6028.255468] LustreError: 3639:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c39bc69e300 x1847765501717120/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6030.180064] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6030.292208] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6034.578803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13409) [ 6034.579115] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13377) [ 6034.609408] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5548 to 0x280000400:5633) [ 6034.609603] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5516 to 0x2c0000400:5601) [ 6034.618768] Lustre: 228913:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8c39b2448e00 x1847765551986944/t55834574850(0) o36->ee6f2298-aae5-4b41-830b-45711a5740a8@192.168.203.20@tcp:186/0 lens 496/2888 e 0 to 0 dl 1762172451 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6034.629962] Lustre: 228913:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6036.206302] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6036.945958] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6037.584161] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6041.486248] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 07:20:22 (1762172422) [ 6045.102715] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6045.826919] Lustre: Failing over lustre-MDT0000 [ 6045.969250] Lustre: server umount lustre-MDT0000 complete [ 6058.182106] Lustre: lustre-MDT0001: Client ee6f2298-aae5-4b41-830b-45711a5740a8 (at 192.168.203.20@tcp) reconnecting [ 6058.185696] Lustre: Skipped 1 previous similar message [ 6060.212658] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6060.214510] LDISKFS-fs (dm-0): recovery complete [ 6060.217863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.565581] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6065.660658] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13409) [ 6065.660706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13441) [ 6066.781186] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6067.333890] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6070.850315] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 07:20:51 (1762172451) [ 6071.217233] LustreError: 230736:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff8c399df65180 x1847765552013312/t60129542149(0) o36->ee6f2298-aae5-4b41-830b-45711a5740a8@192.168.203.20@tcp:197/0 lens 496/456 e 0 to 0 dl 1762172462 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6071.227047] LustreError: 230736:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 5 previous similar messages [ 6073.647798] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6074.272469] Lustre: Failing over lustre-MDT0001 [ 6074.361579] Lustre: server umount lustre-MDT0001 complete [ 6088.438396] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6088.439721] LDISKFS-fs (dm-1): recovery complete [ 6088.443442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6089.873313] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6093.815187] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5516 to 0x2c0000400:5633) [ 6093.815233] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5548 to 0x280000400:5665) [ 6095.153425] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6095.787613] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6099.330799] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 07:21:19 (1762172479) [ 6102.157423] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6104.625082] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6105.290834] Lustre: Failing over lustre-MDT0000 [ 6105.465146] Lustre: server umount lustre-MDT0000 complete [ 6119.704374] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6119.705636] LDISKFS-fs (dm-0): recovery complete [ 6119.710459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6121.295662] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6125.051281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13473) [ 6125.051305] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13441) [ 6126.120741] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6126.629994] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6128.552727] Lustre: Failing over lustre-MDT0001 [ 6128.631517] Lustre: server umount lustre-MDT0001 complete [ 6142.622709] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6142.623807] LDISKFS-fs (dm-1): recovery complete [ 6142.627196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6143.930946] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6148.085877] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5516 to 0x2c0000400:5665) [ 6148.085882] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5548 to 0x280000400:5697) [ 6149.270799] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6149.815811] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6153.305450] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 07:22:13 (1762172533) [ 6156.145794] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6158.798646] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6159.505328] Lustre: Failing over lustre-MDT0000 [ 6159.679646] Lustre: server umount lustre-MDT0000 complete [ 6161.004233] LustreError: 144665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172541 with bad export cookie 16875287510340222553 [ 6161.006078] Lustre: Failing over lustre-MDT0001 [ 6161.009175] LustreError: 144665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6161.116885] Lustre: server umount lustre-MDT0001 complete [ 6176.116825] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6176.117704] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6176.118539] LDISKFS-fs (dm-1): recovery complete [ 6176.119909] LDISKFS-fs (dm-0): recovery complete [ 6176.123989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6176.124290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.200795] LustreError: 239513:0:(llog.c:1616:llog_backup()) MGC192.168.203.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6176.203878] Lustre: 239513:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6179.743141] Lustre: 3642:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762172544/real 1762172544] req@ffff8c39b2451880 x1847765501844992/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1762172560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6185.951520] LustreError: 3639:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c39bfb5ad80 x1847765501848704/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6187.494787] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6187.581750] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6192.179266] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5548 to 0x280000400:5729) [ 6192.179541] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5516 to 0x2c0000400:5697) [ 6192.206364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13123 to 0x280000401:13505) [ 6192.206399] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12958 to 0x2c0000401:13473) [ 6193.359939] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6193.906588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6194.398855] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6197.930678] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 07:22:58 (1762172578) [ 6198.590423] Lustre: 240750:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting ee6f2298-aae5-4b41-830b-45711a5740a8 at adminstrative request [ 6203.397622] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 07:23:03 (1762172583) [ 6205.331884] Lustre: Failing over lustre-MDT0000 [ 6205.585944] Lustre: server umount lustre-MDT0000 complete [ 6218.493974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.984079] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6223.900344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13557 to 0x280000401:13601) [ 6223.900354] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13524 to 0x2c0000401:13569) [ 6225.236182] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6225.875538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6229.501520] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 07:23:30 (1762172610) [ 6233.706518] Lustre: Failing over lustre-OST0000 [ 6233.771251] Lustre: server umount lustre-OST0000 complete [ 6246.860891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6249.259581] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6252.633962] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6253.237371] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6257.184355] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 07:23:57 (1762172637) [ 6258.743687] Lustre: Failing over lustre-MDT0000 [ 6258.903452] Lustre: server umount lustre-MDT0000 complete [ 6262.113817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.950424] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6264.904900] Lustre: lustre-MDT0000: Denying connection for new client 9a28e48b-83cd-48ea-8d7e-74dd61a431ea (at 192.168.203.20@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 6264.912394] Lustre: Skipped 2 previous similar messages [ 6267.383737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13557 to 0x280000401:13633) [ 6267.383960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13524 to 0x2c0000401:13601) [ 6272.737939] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 07:24:13 (1762172653) [ 6276.061501] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6277.031516] Lustre: Failing over lustre-OST0000 [ 6277.068500] Lustre: server umount lustre-OST0000 complete [ 6291.322057] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6291.323317] LDISKFS-fs (dm-2): recovery complete [ 6291.326813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6293.296958] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6296.381836] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6296.936680] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6300.544295] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 07:24:41 (1762172681) [ 6303.343945] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6305.227358] Lustre: Failing over lustre-OST0000 [ 6305.260909] Lustre: server umount lustre-OST0000 complete [ 6319.176230] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6319.177837] LDISKFS-fs (dm-2): recovery complete [ 6319.183303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6321.059824] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.20@tcp inode [0x200031141:0x5:0x0] object 0x280000401:13635 extent [0-4194303]: client csum f9012035, server csum 76945ab9 [ 6321.122587] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6324.067272] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6324.577165] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6327.862062] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 07:25:08 (1762172708) [ 6330.213974] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6332.555266] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6334.432840] Lustre: Failing over lustre-MDT0000 [ 6334.688638] Lustre: server umount lustre-MDT0000 complete [ 6335.872598] LustreError: 6485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762172716 with bad export cookie 16875287510340249321 [ 6335.875073] Lustre: Failing over lustre-OST0000 [ 6335.875785] LustreError: 6485:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6335.916902] Lustre: server umount lustre-OST0000 complete [ 6350.463410] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6350.464737] LDISKFS-fs (dm-0): recovery complete [ 6350.467506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.543481] LustreError: 3639:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c388ca88380 x1847765501991680/t0(0) o250->MGC192.168.203.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6361.965845] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6365.791507] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13524 to 0x2c0000401:13633) [ 6376.265818] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6376.267473] LDISKFS-fs (dm-2): recovery complete [ 6376.271978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6378.409251] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6384.384568] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 07:26:04 (1762172764) [ 6389.254177] Lustre: Failing over lustre-OST0000 [ 6389.307440] Lustre: server umount lustre-OST0000 complete [ 6390.664046] Lustre: Failing over lustre-MDT0000 [ 6390.902415] Lustre: server umount lustre-MDT0000 complete [ 6403.859723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.228904] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6409.206124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13524 to 0x2c0000401:13665) [ 6414.527996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6416.537983] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6417.277900] Lustre: lustre-OST0000: Denying connection for new client bf1973db-bee4-4968-bda2-7b89508ec330 (at 192.168.203.20@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 6417.285449] Lustre: Skipped 1 previous similar message [ 6422.726589] Lustre: lustre-OST0000: Denying connection for new client bf1973db-bee4-4968-bda2-7b89508ec330 (at 192.168.203.20@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 6422.731602] Lustre: Skipped 1 previous similar message [ 6432.966363] Lustre: lustre-OST0000: Denying connection for new client bf1973db-bee4-4968-bda2-7b89508ec330 (at 192.168.203.20@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [ 6432.972420] Lustre: Skipped 1 previous similar message [ 6453.446272] Lustre: lustre-OST0000: Denying connection for new client bf1973db-bee4-4968-bda2-7b89508ec330 (at 192.168.203.20@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:31 [ 6453.451971] Lustre: Skipped 3 previous similar messages [ 6485.000129] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6485.003225] Lustre: 253796:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 9a28e48b-83cd-48ea-8d7e-74dd61a431ea@ [ 6485.008060] Lustre: 253796:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6485.010471] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6485.016697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13684 to 0x280000401:13705) [ 6490.925697] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [ 6495.291796] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 6497.477132] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 07:27:58 (1762172878) [ 6498.950937] Lustre: Failing over lustre-OST0000 [ 6498.981707] Lustre: server umount lustre-OST0000 complete [ 6511.989613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6513.827789] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6517.533909] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 07:28:18 (1762172898) [ 6518.709578] Lustre: Failing over lustre-OST0000 [ 6518.743441] Lustre: server umount lustre-OST0000 complete [ 6531.543264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6533.523937] LustreError: 256779:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6533.528574] LustreError: 256779:0:(ldlm_lib.c:2883:target_recovery_thread()) Skipped 78 previous similar messages [ 6533.584438] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6539.231107] Lustre: *** cfs_fail_loc=715, val=40*** [ 6540.255119] Lustre: *** cfs_fail_loc=715, val=40*** [ 6540.256284] Lustre: Skipped 1 previous similar message [ 6548.447118] Lustre: 3639:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762172913/real 1762172913] req@ffff8c39b78f5f80 x1847765502100096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1762172929 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 6548.456749] Lustre: 3639:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [ 6548.459953] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:53 [ 6548.464473] Lustre: Skipped 1 previous similar message [ 6549.638192] Lustre: lustre-OST0000: Client bf1973db-bee4-4968-bda2-7b89508ec330 (at 192.168.203.20@tcp) reconnected, waiting for 3 clients in recovery for 0:52 [ 6554.591105] Lustre: *** cfs_fail_loc=715, val=40*** [ 6564.831545] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:37 [ 6564.834748] Lustre: Skipped 1 previous similar message [ 6570.975161] Lustre: *** cfs_fail_loc=715, val=40*** [ 6570.976289] Lustre: Skipped 2 previous similar messages [ 6573.575105] LustreError: 256779:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6573.578319] LustreError: 256779:0:(ldlm_lib.c:2883:target_recovery_thread()) Skipped 76 previous similar messages [ 6573.584196] Lustre: lustre-OST0000: Recovery over after 0:41, of 3 clients 3 recovered and 0 were evicted. [ 6573.589052] Lustre: Skipped 19 previous similar messages [ 6574.992662] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6575.635551] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6579.115507] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 07:29:19 (1762172959) [ 6580.342980] Lustre: Failing over lustre-MDT0000 [ 6580.431943] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 6580.434586] Lustre: Skipped 17 previous similar messages [ 6580.559127] LustreError: 257768:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6580.561368] LustreError: 257768:0:(obd_class.h:479:obd_check_dev()) Skipped 139 previous similar messages [ 6580.583435] Lustre: server umount lustre-MDT0000 complete [ 6583.263430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6583.265598] LustreError: Skipped 14 previous similar messages [ 6583.267232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6583.270316] Lustre: Skipped 53 previous similar messages [ 6583.271668] LustreError: 173884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6583.275917] LustreError: 173884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 228 previous similar messages [ 6593.420390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.462943] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6593.467623] LustreError: Skipped 8 previous similar messages [ 6593.556030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6593.558180] Lustre: Skipped 19 previous similar messages [ 6593.588716] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6593.591226] Lustre: Skipped 20 previous similar messages [ 6594.957628] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6595.782792] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6595.785368] Lustre: Skipped 20 previous similar messages [ 6598.627580] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6598.629185] Lustre: Skipped 56 previous similar messages [ 6598.632267] LustreError: 258221:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6602.207167] Lustre: *** cfs_fail_loc=715, val=80*** [ 6602.209511] Lustre: Skipped 2 previous similar messages [ 6611.142162] Lustre: lustre-MDT0000: Client bf1973db-bee4-4968-bda2-7b89508ec330 (at 192.168.203.20@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 6611.145285] Lustre: Skipped 1 previous similar message [ 6615.007364] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6617.567105] Lustre: *** cfs_fail_loc=715, val=80*** [ 6617.568472] Lustre: Skipped 1 previous similar message [ 6627.535386] Lustre: lustre-MDT0000: Client bf1973db-bee4-4968-bda2-7b89508ec330 (at 192.168.203.20@tcp) reconnected, waiting for 2 clients in recovery for 1:11 [ 6630.367509] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6633.951112] Lustre: *** cfs_fail_loc=715, val=80*** [ 6633.952397] Lustre: Skipped 1 previous similar message [ 6643.910140] Lustre: lustre-MDT0000: Client bf1973db-bee4-4968-bda2-7b89508ec330 (at 192.168.203.20@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 6646.751399] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6660.294178] Lustre: lustre-MDT0000: Client bf1973db-bee4-4968-bda2-7b89508ec330 (at 192.168.203.20@tcp) reconnected, waiting for 2 clients in recovery for 0:38 [ 6663.135520] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6666.719153] Lustre: *** cfs_fail_loc=715, val=80*** [ 6666.720475] Lustre: Skipped 3 previous similar messages [ 6678.495158] Lustre: 3639:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1762173043/real 1762173043] req@ffff8c39be058a80 x1847765502165248/t0(0) o400->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 1 dl 1762173059 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 6678.502500] Lustre: 3639:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 6678.506246] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6678.711087] LustreError: 258221:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6678.715728] Lustre: 258221:0:(ldlm_lib.c:2929:target_recovery_thread()) too long recovery - read logs [ 6678.717993] LustreError: dumping log to /tmp/lustre-log.1762173059.258221 [ 6678.775226] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13678 to 0x2c0000401:13697) [ 6678.775237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13719 to 0x280000401:13737) [ 6679.978329] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6680.475628] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6683.977845] Lustre: DEBUG MARKER: == replay-single test complete, duration 6520 sec ======== 07:31:04 (1762173064) [ 6684.506658] Lustre: DEBUG MARKER: === replay-single: start cleanup 07:31:05 (1762173065) === [ 6686.971571] Lustre: DEBUG MARKER: === replay-single: finish cleanup 07:31:07 (1762173067) === [ 6723.340385] Lustre: server umount lustre-MDT0000 complete [ 6725.914680] LustreError: 42030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1762173106 with bad export cookie 16875287510340262012 [ 6725.918250] LustreError: 42030:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6726.012715] Lustre: server umount lustre-MDT0001 complete [ 6738.939640] Lustre: server umount lustre-OST0000 complete [ 6751.790432] Lustre: server umount lustre-OST0001 complete [ 6757.371687] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 6758.503773] Key type lgssc unregistered [ 6758.625461] LNet: 261383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6758.628237] LNetError: 261383:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6758.636431] LNet: Removed LNI 192.168.203.120@tcp [ 6758.993163] Key type .llcrypt unregistered [ 6758.994206] Key type ._llcrypt unregistered