[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 567865547 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003349] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007962] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010144] LSM: Security Framework initializing [ 0.011062] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013066] *** VALIDATE selinux *** [ 0.022328] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027631] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028186] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030141] *** VALIDATE tmpfs *** [ 0.032346] *** VALIDATE proc *** [ 0.033295] *** VALIDATE cgroup *** [ 0.034013] *** VALIDATE cgroup2 *** [ 0.035307] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039014] Speculative Store Bypass: Vulnerable [ 0.042758] debug: unmapping init [mem 0xffffffffb0059000-0xffffffffb0060fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045797] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047008] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052015] ... event mask: 000000070000000f [ 0.054277] rcu: Hierarchical SRCU implementation. [ 0.056443] smp: Bringing up secondary CPUs ... [ 0.057635] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.068181] smp: Brought up 1 node, 4 CPUs [ 0.070014] smpboot: Max logical packages: 1 [ 0.071013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106183] node 0 deferred pages initialised in 33ms [ 0.110035] devtmpfs: initialized [ 0.111271] x86/mm: Memory block size: 128MB [ 0.113899] gcov: version magic: 0x41383552 [ 0.115389] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.116123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.118108] pinctrl core: initialized pinctrl subsystem [ 0.120214] [ 0.120922] ************************************************************* [ 0.124015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.126012] ** ** [ 0.129015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.131013] ** ** [ 0.133015] ** This means that this kernel is built to expose internal ** [ 0.135013] ** IOMMU data structures, which may compromise security on ** [ 0.137014] ** your system. ** [ 0.139014] ** ** [ 0.141012] ** If you see this message and you are not debugging the ** [ 0.143014] ** kernel, report this immediately to your vendor! ** [ 0.145012] ** ** [ 0.148019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150014] ************************************************************* [ 0.152000] NET: Registered protocol family 16 [ 0.154179] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.156078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.159075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.164008] cpuidle: using governor menu [ 0.166000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170126] PCI: Using configuration type 1 for base access [ 0.172129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.182382] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.183030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186171] cryptd: max_cpu_qlen set to 1000 [ 0.188234] ACPI: Added _OSI(Module Device) [ 0.189000] ACPI: Added _OSI(Processor Device) [ 0.189012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.195942] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206022] ACPI: Interpreter enabled [ 0.207080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208014] ACPI: Using IOAPIC for interrupt routing [ 0.209134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210481] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223105] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.230091] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240800] acpiphp: Slot [2] registered [ 0.243237] acpiphp: Slot [5] registered [ 0.245160] acpiphp: Slot [6] registered [ 0.247187] acpiphp: Slot [7] registered [ 0.249221] acpiphp: Slot [8] registered [ 0.251150] acpiphp: Slot [9] registered [ 0.253218] acpiphp: Slot [10] registered [ 0.256949] acpiphp: Slot [3] registered [ 0.259141] acpiphp: Slot [4] registered [ 0.262152] acpiphp: Slot [11] registered [ 0.264126] acpiphp: Slot [12] registered [ 0.266112] acpiphp: Slot [13] registered [ 0.268111] acpiphp: Slot [14] registered [ 0.269095] acpiphp: Slot [15] registered [ 0.271103] acpiphp: Slot [16] registered [ 0.273103] acpiphp: Slot [17] registered [ 0.275134] acpiphp: Slot [18] registered [ 0.277118] acpiphp: Slot [19] registered [ 0.279118] acpiphp: Slot [20] registered [ 0.280149] acpiphp: Slot [21] registered [ 0.282089] acpiphp: Slot [22] registered [ 0.283226] acpiphp: Slot [23] registered [ 0.286000] acpiphp: Slot [24] registered [ 0.287639] acpiphp: Slot [25] registered [ 0.290106] acpiphp: Slot [26] registered [ 0.291189] acpiphp: Slot [27] registered [ 0.293243] acpiphp: Slot [28] registered [ 0.295097] acpiphp: Slot [29] registered [ 0.297190] acpiphp: Slot [30] registered [ 0.300145] acpiphp: Slot [31] registered [ 0.301000] PCI host bridge to bus 0000:00 [ 0.301000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.301000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.301000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.301000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.301000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.301000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.301000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.301000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.301000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.301000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.301000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.301000] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.301000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.301000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.301000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.301000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.301000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.301000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.301000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.301000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.301000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.301000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.301000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.301000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.301000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.301000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.301000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.301000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.301000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.301000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.301000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.301000] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.301000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.301000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.301000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.301000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.301000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.301000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.301000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.301000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.301000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.301000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.301000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.301000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.301000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.301000] iommu: Default domain type: Passthrough [ 0.301000] SCSI subsystem initialized [ 0.301000] ACPI: bus type USB registered [ 0.301000] usbcore: registered new interface driver usbfs [ 0.301000] usbcore: registered new interface driver hub [ 0.323108] usbcore: registered new device driver usb [ 0.345228] pps_core: LinuxPPS API ver. 1 registered [ 0.347024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.349105] PTP clock support registered [ 0.351218] EDAC MC: Ver: 3.0.0 [ 0.355165] PCI: Using ACPI for IRQ routing [ 0.357200] NetLabel: Initializing [ 0.358013] NetLabel: domain hash size = 128 [ 0.359007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.361101] NetLabel: unlabeled traffic allowed by default [ 0.365076] vgaarb: loaded [ 0.367432] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.375015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.382565] clocksource: Switched to clocksource kvm-clock [ 0.691247] VFS: Disk quotas dquot_6.6.0 [ 0.693281] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.696348] *** VALIDATE ramfs *** [ 0.698168] *** VALIDATE hugetlbfs *** [ 0.700299] pnp: PnP ACPI init [ 0.703596] pnp: PnP ACPI: found 6 devices [ 0.726373] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.730187] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.733485] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.736146] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.739436] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.743175] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.747433] NET: Registered protocol family 2 [ 0.751443] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.759263] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.797577] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.802972] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.806546] TCP: Hash tables configured (established 65536 bind 65536) [ 0.809488] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.812652] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.815587] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.818654] NET: Registered protocol family 1 [ 0.826863] RPC: Registered named UNIX socket transport module. [ 0.829674] RPC: Registered udp transport module. [ 0.832110] RPC: Registered tcp transport module. [ 0.834232] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.837071] NET: Registered protocol family 44 [ 0.839221] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.841666] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.844179] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.846973] PCI: CLS 0 bytes, default 64 [ 0.849183] Unpacking initramfs... [ 2.345461] hrtimer: interrupt took 27270877 ns [ 4.251371] debug: unmapping init [mem 0xffff92e3bcc54000-0xffff92e3bffbffff] [ 4.277826] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.287429] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.297452] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.484394] Initialise system trusted keyrings [ 6.487193] Key type blacklist registered [ 6.619098] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.660937] zbud: loaded [ 6.675950] *** VALIDATE nfs *** [ 6.683782] *** VALIDATE nfs4 *** [ 6.733420] pstore: using deflate compression [ 6.799253] Platform Keyring initialized [ 7.036720] NET: Registered protocol family 38 [ 7.039283] Key type asymmetric registered [ 7.040686] Asymmetric key parser 'x509' registered [ 7.042179] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.045704] io scheduler mq-deadline registered [ 7.047669] io scheduler kyber registered [ 7.049345] io scheduler bfq registered [ 7.051211] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.054319] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.057096] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.059988] ACPI: Power Button [PWRF] [ 7.074915] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.089038] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.105895] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.117758] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.138303] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.169180] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.214467] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.221159] Non-volatile memory driver v1.3 [ 7.222770] Linux agpgart interface v0.103 [ 7.258586] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 7.261022] vda: detected capacity change from 0 to 68624384 [ 7.274492] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.277254] vdb: detected capacity change from 0 to 1073741824 [ 7.292834] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.295697] vdc: detected capacity change from 0 to 2621440000 [ 7.312404] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 7.314638] vdd: detected capacity change from 0 to 2621440000 [ 7.328114] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.330542] vde: detected capacity change from 0 to 4294967296 [ 7.369363] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 7.373255] vdf: detected capacity change from 0 to 4294967296 [ 7.418661] libphy: Fixed MDIO Bus: probed [ 7.465434] usbcore: registered new interface driver usbserial_generic [ 7.467664] usbserial: USB Serial support registered for generic [ 7.470360] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.474829] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.476308] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.478435] mousedev: PS/2 mouse device common for all mice [ 7.482441] rtc_cmos 00:05: RTC can wake from S4 [ 7.485097] rtc_cmos 00:05: registered as rtc0 [ 7.486918] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.489426] intel_pstate: CPU model not supported [ 7.492524] hid: raw HID events driver (C) Jiri Kosina [ 7.494300] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.497637] usbcore: registered new interface driver usbhid [ 7.500015] usbhid: USB HID core driver [ 7.501438] drop_monitor: Initializing network drop monitor service [ 7.503721] Initializing XFRM netlink socket [ 7.505381] NET: Registered protocol family 10 [ 7.509433] Segment Routing with IPv6 [ 7.511839] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.512372] NET: Registered protocol family 17 [ 7.517044] mpls_gso: MPLS GSO support [ 7.521516] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.526301] RAS: Correctable Errors collector initialized. [ 7.528067] AVX version of gcm_enc/dec engaged. [ 7.529635] AES CTR mode by8 optimization enabled [ 7.631973] sched_clock: Marking stable (7631892015, 0)->(9350090332, -1718198317) [ 7.639732] registered taskstats version 1 [ 7.643349] Loading compiled-in X.509 certificates [ 7.647621] zswap: loaded using pool lzo/zbud [ 7.725441] Key type big_key registered [ 7.754526] Key type encrypted registered [ 7.756442] ima: No TPM chip found, activating TPM-bypass! [ 7.758873] ima: Allocated hash algorithm: sha1 [ 7.761319] ima: No architecture policies found [ 7.763394] evm: Initialising EVM extended attributes: [ 7.765634] evm: security.selinux [ 7.767445] evm: security.ima [ 7.768827] evm: security.capability [ 7.770353] evm: HMAC attrs: 0x1 [ 7.903250] rtc_cmos 00:05: setting system clock to 2026-01-02 00:49:29 UTC (1767314969) [ 7.911789] debug: unmapping init [mem 0xffffffffb1003000-0xffffffffb11fffff] [ 8.157614] debug: unmapping init [mem 0xffffffffafd82000-0xffffffffb0058fff] [ 8.167124] Write protecting the kernel read-only data: 28672k [ 8.171746] debug: unmapping init [mem 0xffffffffae403000-0xffffffffae5fffff] [ 8.175397] debug: unmapping init [mem 0xffffffffaed14000-0xffffffffaedfffff] [ 8.230852] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.238408] systemd[1]: Detected virtualization kvm. [ 8.240172] systemd[1]: Detected architecture x86-64. [ 8.242228] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.269609] systemd[1]: No hostname configured. [ 8.271098] systemd[1]: Set hostname to . [ 8.272783] random: systemd: uninitialized urandom read (16 bytes read) [ 8.275104] systemd[1]: Initializing machine ID from random generator. [ 8.464652] random: ln: uninitialized urandom read (6 bytes read) [ 8.983642] random: systemd: uninitialized urandom read (16 bytes read) [ 8.992862] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 9.009325] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 9.014415] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Journal Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.797506] device-mapper: uevent: version 1.0.3 [ 10.800672] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted Kernel Configuration File System. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 12.334003] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.646068] virtio_net virtio0 ens2: renamed from eth0 [ 14.706591] scsi host0: ata_piix [ 14.727736] scsi host1: ata_piix [ 14.735900] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.748343] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 25.995347] random: crng init done [ 26.000789] random: 7 urandom warning(s) missed due to ratelimiting [ 28.750651] dracut-initqueue[568]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 31.494805] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 33.977254] printk: systemd: 17 output lines suppressed due to ratelimiting [ 34.867847] SELinux: Disabled at runtime. [ 35.046869] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 35.065207] systemd[1]: Detected virtualization kvm. [ 35.069234] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 37.898619] systemd[1]: initrd-switch-root.service: Succeeded. [ 37.902780] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 37.912265] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 37.947787] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 37.970696] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 38.037446] systemd[1]: Starting Journal Service... Starting Journal Service... [ 38.082726] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 38.753603] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 40.053749] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ 40.085283] systemd[1]: Reached target RPC Port Mapper. [ OK ] Reached target RPC Port Mapper. [ 40.145700] systemd[1]: sysroot-var-lib-nfs-rpc_pipefs.mount: Succeeded. [ 40.177787] systemd[1]: sysroot.mount: Succeeded. [ 40.216727] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 41.463640] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 43.755536] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 44.192338] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 44.685592] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 44.916350] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (10s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 51.513416] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit)[ 52.426610] NFS: Registering the id_resolver key type [ 52.436237] Key type id_resolver registered [ 52.441619] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg323-server login: [ 117.358482] libcfs: loading out-of-tree module taints kernel. [ 117.410552] Key type ._llcrypt registered [ 117.416520] Key type .llcrypt registered [ 117.510983] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_hostid [ 194.850706] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 196.218262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 196.229692] alg: No test for adler32 (adler32-zlib) [ 197.638700] Lustre: Lustre: Build Version: 2.17.0_RC4_1_g3742a8b [ 198.608454] LNet: Added LNI 192.168.203.123@tcp [8/256/0/180] [ 200.351157] Key type lgssc registered [ 201.951725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 228.922729] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 230.540320] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 245.346092] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 255.450485] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 266.207071] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 353.284753] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing load_modules_local [ 375.491114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 375.570703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 375.596778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 376.919119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 376.944578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 377.043257] Lustre: lustre-MDT0000: new disk, initializing [ 377.148216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 377.170045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 381.636107] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 390.175213] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 408.532483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 408.613994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 408.748261] Lustre: 14428:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 408.807167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 408.821075] Lustre: Skipped 1 previous similar message [ 408.978276] Lustre: lustre-MDT0001: new disk, initializing [ 409.053850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 409.076913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 409.098455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 413.409673] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 420.857621] Lustre: 16490:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 439.798454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 439.918723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 440.347773] Lustre: lustre-OST0000: new disk, initializing [ 440.360275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 440.483894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.069159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 446.103458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 446.192190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.379357] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 471.661281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 471.716479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 471.782739] Lustre: lustre-OST0001: new disk, initializing [ 471.791081] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 471.846615] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 477.203245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 477.215696] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 477.266855] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 477.808467] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 489.543384] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 495.623966] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 508.273864] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing check_logdir /tmp/testlogs/ [ 514.549834] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing yml_node [ 524.091581] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 527.478720] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 530.189187] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 532.087333] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Thu Jan 1 19:58:11 EST 2026 [ 552.960918] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 555.317927] Lustre: DEBUG MARKER: === replay-single: start setup 19:58:34 (1767315514) === [ 559.603132] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing check_config_client /mnt/lustre [ 606.187398] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 609.931516] Lustre: 23870:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 614.679977] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 620.601742] Lustre: DEBUG MARKER: === replay-single: finish setup 19:59:40 (1767315580) === [ 623.267977] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 19:59:42 (1767315582) [ 632.588570] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 634.851280] Lustre: Failing over lustre-MDT0000 [ 635.021289] LustreError: 24742:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 635.054419] LustreError: 24742:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff92e405547c00) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 635.278439] Lustre: server umount lustre-MDT0000 complete [ 635.886354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.917849] Lustre: Skipped 1 previous similar message [ 640.997831] LustreError: 14440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 641.036484] LustreError: 14440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 642.883425] LustreError: 14435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.090742] LustreError: 14440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 644.112448] LustreError: 14440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 648.001240] LustreError: 14435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.261227] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315597/real 1767315597] req@ffff92e30353e680 x1853164261347840/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315613 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 652.286676] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 652.301794] LustreError: 23251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 652.314148] LustreError: 23251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 661.473653] LustreError: 14435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.494643] LustreError: 14435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 672.920176] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 672.921917] LDISKFS-fs (dm-0): recovery complete [ 673.043724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.831425] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e43fa3b800 x1853164261359360/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 677.470367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.565975] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 679.754970] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 682.495043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 682.621577] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 683.769831] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 691.707385] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 693.778128] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 704.658195] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 20:01:04 (1767315664) [ 707.438910] Lustre: Failing over lustre-OST0000 [ 707.483090] LustreError: 26480:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 707.487032] LustreError: 26480:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 707.558473] Lustre: server umount lustre-OST0000 complete [ 707.560575] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 707.578488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 707.598199] Lustre: Skipped 2 previous similar messages [ 707.613147] LustreError: 17338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.632897] LustreError: 17338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 731.596069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 731.806749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 731.826648] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 732.949530] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 733.905314] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 733.926797] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 733.949297] Lustre: Skipped 3 previous similar messages [ 739.462373] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 749.174231] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 751.307696] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 762.465906] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 20:02:02 (1767315722) [ 771.855641] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 774.814281] Lustre: Failing over lustre-MDT0000 [ 774.989313] LustreError: 28333:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 774.994693] LustreError: 28333:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 775.137759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 775.141043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.161738] Lustre: Skipped 1 previous similar message [ 775.167030] LustreError: 14440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 775.179503] LustreError: 14440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 775.209689] Lustre: server umount lustre-MDT0000 complete [ 778.214234] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 778.231039] Lustre: Skipped 2 previous similar messages [ 793.504503] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315739/real 1767315739] req@ffff92e436e9ce00 x1853164261421824/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315755 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 793.554628] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 809.154915] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 809.161610] LDISKFS-fs (dm-0): recovery complete [ 809.175693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.180802] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c22563 [ 819.196796] Lustre: MGC192.168.203.123@tcp: Connection restored to 0@lo (at 0@lo) [ 819.211832] Lustre: Skipped 1 previous similar message [ 819.514499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.572629] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 824.306098] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 824.800698] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 824.805939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 827.034356] Lustre: lustre-MDT0000: Denying connection for new client e491ab3e-70c9-4dfa-a040-a0a976cf2c81 (at 192.168.203.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 832.322394] Lustre: lustre-MDT0000: Denying connection for new client e491ab3e-70c9-4dfa-a040-a0a976cf2c81 (at 192.168.203.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 837.449854] Lustre: lustre-MDT0000: Denying connection for new client e491ab3e-70c9-4dfa-a040-a0a976cf2c81 (at 192.168.203.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 842.561314] Lustre: lustre-MDT0000: Denying connection for new client e491ab3e-70c9-4dfa-a040-a0a976cf2c81 (at 192.168.203.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:51 [ 847.689524] Lustre: lustre-MDT0000: Denying connection for new client e491ab3e-70c9-4dfa-a040-a0a976cf2c81 (at 192.168.203.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:46 [ 857.923109] Lustre: lustre-MDT0000: Denying connection for new client e491ab3e-70c9-4dfa-a040-a0a976cf2c81 (at 192.168.203.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:36 [ 857.954127] Lustre: Skipped 1 previous similar message [ 878.427043] Lustre: lustre-MDT0000: Denying connection for new client e491ab3e-70c9-4dfa-a040-a0a976cf2c81 (at 192.168.203.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:16 [ 878.444420] Lustre: Skipped 3 previous similar messages [ 894.501070] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 894.505377] Lustre: 29278:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d2bdb49c-cefb-4661-a489-5dfe88ad913e@ [ 894.539637] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 894.575381] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 894.575662] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 894.603902] Lustre: Skipped 2 previous similar messages [ 894.611544] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 894.612340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 907.893295] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 20:04:27 (1767315867) [ 915.843775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 917.975171] Lustre: Failing over lustre-MDT0000 [ 918.073899] LustreError: 30351:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 918.085576] LustreError: 30351:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 918.197967] Lustre: server umount lustre-MDT0000 complete [ 920.544046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 920.547760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 920.561344] LustreError: 14439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.583471] LustreError: 14439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 36 previous similar messages [ 937.442712] Lustre: 6800:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767315883/real 1767315883] req@ffff92e411191c00 x1853164261485824/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767315899 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 937.480789] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 947.511893] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 947.514593] LDISKFS-fs (dm-0): recovery complete [ 947.525526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.412978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.460743] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 968.027518] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 968.677623] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 968.692271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 971.554238] Lustre: lustre-MDT0000: Denying connection for new client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 971.569900] Lustre: Skipped 3 previous similar messages [ 1038.143571] Lustre: lustre-MDT0000: Denying connection for new client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:00 [ 1038.176081] Lustre: Skipped 12 previous similar messages [ 1038.500905] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1038.506088] Lustre: 31292:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client e491ab3e-70c9-4dfa-a040-a0a976cf2c81@ [ 1038.521716] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1038.554967] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 1038.558873] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1038.558881] Lustre: Skipped 2 previous similar messages [ 1038.647036] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 1038.647232] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 1053.615841] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 20:06:52 (1767316012) [ 1062.025995] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1064.620396] Lustre: Failing over lustre-MDT0000 [ 1064.839887] LustreError: 32350:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1064.848290] LustreError: 32350:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1065.028554] Lustre: server umount lustre-MDT0000 complete [ 1065.952095] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.962164] LustreError: 23251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.976657] Lustre: Skipped 5 previous similar messages [ 1066.024285] LustreError: 23251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 39 previous similar messages [ 1082.344947] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316027/real 1767316027] req@ffff92e40656c380 x1853164261546368/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316043 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1082.384151] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1095.052297] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1095.054892] LDISKFS-fs (dm-0): recovery complete [ 1095.061767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1106.914097] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e42b4a1f80 x1853164261559552/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1107.248419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.295345] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1109.072154] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1112.404294] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1112.565194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1112.690922] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1112.750456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 1112.755921] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 1121.087540] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1122.926305] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1133.209536] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 20:08:12 (1767316092) [ 1142.425622] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1144.821721] Lustre: Failing over lustre-MDT0000 [ 1145.022427] LustreError: 34470:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1145.031984] LustreError: 34470:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1145.215782] Lustre: server umount lustre-MDT0000 complete [ 1148.386177] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.395878] Lustre: Skipped 3 previous similar messages [ 1163.731085] Lustre: 6797:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316109/real 1767316109] req@ffff92e4344bb480 x1853164261587712/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1163.762688] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1174.974337] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1174.981196] LDISKFS-fs (dm-0): recovery complete [ 1174.998299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.401421] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e4344bb480 x1853164261600896/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1190.750326] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1191.765892] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1195.694254] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1196.023742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1196.036649] Lustre: Skipped 3 previous similar messages [ 1196.091817] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1196.130552] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 1196.134743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1203.557583] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1205.417925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1215.164565] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 20:09:34 (1767316174) [ 1224.055676] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1226.147087] Lustre: Failing over lustre-MDT0000 [ 1226.320760] LustreError: 36587:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1226.333963] LustreError: 36587:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1226.417855] Lustre: server umount lustre-MDT0000 complete [ 1226.728625] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1226.733841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1226.750416] Lustre: Skipped 3 previous similar messages [ 1248.225230] Lustre: 6799:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316193/real 1767316193] req@ffff92e436d94000 x1853164261627904/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1248.254922] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1256.897315] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1256.900366] LDISKFS-fs (dm-0): recovery complete [ 1256.911401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.730191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.734827] Lustre: Skipped 1 previous similar message [ 1257.794691] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1258.309746] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1262.918824] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1263.249595] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1263.303665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 1263.309277] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 1271.028315] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1272.754562] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1281.673655] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 20:10:41 (1767316241) [ 1289.001640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1290.811492] Lustre: Failing over lustre-MDT0000 [ 1290.929749] LustreError: 38702:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1290.934664] LustreError: 38702:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1291.002662] Lustre: server umount lustre-MDT0000 complete [ 1293.792328] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.801809] Lustre: Skipped 4 previous similar messages [ 1310.179618] Lustre: 6800:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316255/real 1767316255] req@ffff92e433b51500 x1853164261662592/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316271 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1310.213115] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1318.906892] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1318.909600] LDISKFS-fs (dm-0): recovery complete [ 1318.921514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.096206] LustreError: 39613:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1320.103308] LustreError: 39613:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff92e433ab6680 x1853164261668480/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767316281 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1320.144239] LustreError: 39613:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1320.758317] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1322.572910] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1326.020971] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1326.062047] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1326.067196] Lustre: Skipped 7 previous similar messages [ 1326.255436] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1326.336701] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 1326.336852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1337.048543] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1339.589995] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1349.196995] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 20:11:48 (1767316308) [ 1357.756813] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1359.676097] Lustre: Failing over lustre-MDT0000 [ 1359.817317] LustreError: 40821:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1359.819284] LustreError: 40821:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1360.115459] Lustre: server umount lustre-MDT0000 complete [ 1361.904621] LustreError: 14434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.918446] LustreError: 14434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 169 previous similar messages [ 1378.275381] Lustre: 6800:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316323/real 1767316323] req@ffff92e433132a00 x1853164261701120/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316339 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1378.320687] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1388.599223] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1388.604283] LDISKFS-fs (dm-0): recovery complete [ 1388.619248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.113677] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1404.236444] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1407.309547] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1408.586706] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1408.623132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1408.623776] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1415.261449] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1416.757197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1426.495032] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 20:13:06 (1767316386) [ 1427.776464] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1427.784444] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1427.790615] LustreError: 14438:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e433130e00 x1853164180632960/t38654705666(0) o35->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:320/0 lens 392/456 e 0 to 0 dl 1767316400 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1437.504880] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1439.657814] Lustre: Failing over lustre-MDT0000 [ 1439.734606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1439.746365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1439.765110] Lustre: Skipped 4 previous similar messages [ 1439.772645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1440.004087] Lustre: server umount lustre-MDT0000 complete [ 1459.492035] Lustre: 6800:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316405/real 1767316405] req@ffff92e433aa5c00 x1853164261745280/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316421 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1459.518757] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1469.754310] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 1469.756186] LDISKFS-fs (dm-0): recovery complete [ 1469.767640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.675855] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1475.713813] Lustre: 14438:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e436d96d80 x1853164180632960/t38654705666(0) o35->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:368/0 lens 392/456 e 0 to 0 dl 1767316448 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 1475.745282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1475.752378] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1482.714429] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1484.400393] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1493.560863] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 20:14:13 (1767316453) [ 1502.120915] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1504.031266] Lustre: Failing over lustre-MDT0000 [ 1504.160105] LustreError: 45100:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1504.163283] LustreError: 45100:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 1504.372524] Lustre: server umount lustre-MDT0000 complete [ 1535.158341] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1535.160851] LDISKFS-fs (dm-0): recovery complete [ 1535.167633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1547.233353] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e302360e00 x1853164261790464/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1547.615105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.621690] Lustre: Skipped 3 previous similar messages [ 1547.663858] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1547.675450] Lustre: Skipped 1 previous similar message [ 1548.960948] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1548.978493] Lustre: Skipped 1 previous similar message [ 1552.212313] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1552.946669] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1552.960268] Lustre: Skipped 1 previous similar message [ 1553.024368] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 1553.025689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 1560.636937] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1562.483587] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1571.697508] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 20:15:31 (1767316531) [ 1580.521832] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1581.846547] Lustre: *** cfs_fail_loc=114, val=0*** [ 1585.070266] Lustre: Failing over lustre-MDT0000 [ 1585.454091] Lustre: server umount lustre-MDT0000 complete [ 1588.711885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1604.562121] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316550/real 1767316550] req@ffff92e417525500 x1853164261819008/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1604.593630] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1604.608380] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1604.622084] LustreError: Skipped 1 previous similar message [ 1614.723763] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1614.726250] LDISKFS-fs (dm-0): recovery complete [ 1614.734816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1635.433512] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1635.828988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1635.836115] Lustre: Skipped 15 previous similar messages [ 1635.922475] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 1635.924796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 1643.883435] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1645.843228] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1656.013607] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 20:16:55 (1767316615) [ 1665.254525] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1666.611059] Lustre: *** cfs_fail_loc=128, val=0*** [ 1670.185911] Lustre: Failing over lustre-MDT0000 [ 1670.563363] Lustre: server umount lustre-MDT0000 complete [ 1701.434444] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1701.436777] LDISKFS-fs (dm-0): recovery complete [ 1701.447592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1713.130420] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c25601 [ 1718.584691] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1718.967244] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 1718.967777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 1727.137400] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1729.238116] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1738.539302] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 20:18:18 (1767316698) [ 1745.563784] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1747.892481] Lustre: Failing over lustre-MDT0000 [ 1748.156785] Lustre: server umount lustre-MDT0000 complete [ 1749.473173] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1749.489225] Lustre: Skipped 17 previous similar messages [ 1778.039740] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1778.042419] LDISKFS-fs (dm-0): recovery complete [ 1778.050714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.496029] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e433a19880 x1853164261907584/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1795.393682] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1796.469394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 1796.469653] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 1802.908708] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1805.041849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1816.680650] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 20:19:35 (1767316775) [ 1826.663724] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1828.701652] Lustre: Failing over lustre-MDT0000 [ 1828.888522] LustreError: 53766:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1828.902713] LustreError: 53766:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 1829.151405] Lustre: server umount lustre-MDT0000 complete [ 1857.422329] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1857.425400] LDISKFS-fs (dm-0): recovery complete [ 1857.432650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1873.158717] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1873.171924] Lustre: Skipped 3 previous similar messages [ 1875.214568] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1875.225763] Lustre: Skipped 3 previous similar messages [ 1878.183285] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1878.691848] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1878.700389] Lustre: Skipped 3 previous similar messages [ 1878.762354] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 1878.766230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 1886.498712] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1888.119344] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1897.986747] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 20:20:57 (1767316857) [ 1905.785745] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1913.839590] Lustre: Failing over lustre-MDT0000 [ 1914.205395] Lustre: server umount lustre-MDT0000 complete [ 1914.351478] LustreError: 16316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.375524] LustreError: 16316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 311 previous similar messages [ 1929.695157] Lustre: 6800:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767316875/real 1767316875] req@ffff92e42b911c00 x1853164261981952/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767316891 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1929.740345] Lustre: 6800:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 1929.756070] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1929.764930] LustreError: Skipped 3 previous similar messages [ 1942.472211] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1942.475851] LDISKFS-fs (dm-0): recovery complete [ 1942.484198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.601987] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1964.061702] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 1964.062898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 1967.704970] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1969.285893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1992.581825] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 20:22:32 (1767316952) [ 1999.248252] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2001.076737] Lustre: Failing over lustre-MDT0000 [ 2003.294916] Lustre: server umount lustre-MDT0000 complete [ 2031.735335] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2031.739737] LDISKFS-fs (dm-0): recovery complete [ 2031.747855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.608384] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e42bb45c00 x1853164262090496/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2037.550473] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2038.390099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 2038.390252] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 2046.188495] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2047.823807] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2059.666361] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 20:23:39 (1767317019) [ 2068.296269] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2070.331371] Lustre: Failing over lustre-MDT0000 [ 2070.359131] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 2070.740958] Lustre: server umount lustre-MDT0000 complete [ 2074.081344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2099.009760] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2099.014470] LDISKFS-fs (dm-0): recovery complete [ 2099.026368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2100.726532] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e42ba96a00 x1853164262137216/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2100.947683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.951316] Lustre: Skipped 6 previous similar messages [ 2104.780900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2106.412474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 2106.419674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 2112.459730] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2114.694366] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2125.864363] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 20:24:45 (1767317085) [ 2134.005089] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2135.907874] Lustre: Failing over lustre-MDT0000 [ 2136.132686] Lustre: server umount lustre-MDT0000 complete [ 2137.069955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2165.505677] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2165.509105] LDISKFS-fs (dm-0): recovery complete [ 2165.518106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2183.282817] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2183.681276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2183.693279] Lustre: Skipped 28 previous similar messages [ 2183.838239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 2183.845499] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 2191.084873] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2193.123993] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2201.989290] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 20:26:01 (1767317161) [ 2209.641191] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2211.884730] Lustre: Failing over lustre-MDT0000 [ 2212.134799] Lustre: server umount lustre-MDT0000 complete [ 2241.482996] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2241.486323] LDISKFS-fs (dm-0): recovery complete [ 2241.498288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.153267] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2261.120813] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 2261.121567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 2268.775895] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2270.528992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2280.121779] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 20:27:19 (1767317239) [ 2288.110785] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2290.084834] Lustre: Failing over lustre-MDT0000 [ 2290.389979] Lustre: server umount lustre-MDT0000 complete [ 2291.682141] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.693361] Lustre: Skipped 27 previous similar messages [ 2291.694429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2320.450483] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2320.454288] LDISKFS-fs (dm-0): recovery complete [ 2320.472617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.634970] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2338.378930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 2338.380748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 2344.947880] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2346.703706] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2355.333489] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 20:28:35 (1767317315) [ 2363.352523] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2365.488178] Lustre: Failing over lustre-MDT0000 [ 2365.674357] LustreError: 68627:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2365.679874] LustreError: 68627:0:(obd_class.h:479:obd_check_dev()) Skipped 55 previous similar messages [ 2365.866179] Lustre: server umount lustre-MDT0000 complete [ 2394.305215] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2394.309706] LDISKFS-fs (dm-0): recovery complete [ 2394.320890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2394.921757] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2394.933835] Lustre: Skipped 6 previous similar messages [ 2396.997228] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2397.003874] Lustre: Skipped 6 previous similar messages [ 2400.119892] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2400.387776] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2400.399466] Lustre: Skipped 6 previous similar messages [ 2400.484856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 2400.486120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 2408.404663] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2410.473203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2421.138033] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 20:29:40 (1767317380) [ 2428.157237] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2430.089394] Lustre: Failing over lustre-MDT0000 [ 2430.405964] Lustre: server umount lustre-MDT0000 complete [ 2451.423765] Lustre: 6797:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317397/real 1767317397] req@ffff92e417a57800 x1853164262311680/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767317413 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2451.465799] Lustre: 6797:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 2451.475111] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2451.483447] LustreError: Skipped 6 previous similar messages [ 2459.672312] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2459.674347] LDISKFS-fs (dm-0): recovery complete [ 2459.680784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2461.791592] LustreError: 71652:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2461.797038] LustreError: 71652:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff92e42b6a3b80 x1853164262317056/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767317423 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2461.812580] LustreError: 71652:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2462.710741] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c323ee [ 2467.665573] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2468.537304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 2468.544664] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 2476.200887] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2478.370585] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2487.404744] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 20:30:47 (1767317447) [ 2496.150673] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2498.623798] Lustre: Failing over lustre-MDT0000 [ 2498.855311] Lustre: server umount lustre-MDT0000 complete [ 2514.402266] LustreError: 70114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2514.440222] LustreError: 70114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 342 previous similar messages [ 2528.188716] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2528.195607] LDISKFS-fs (dm-0): recovery complete [ 2528.224479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.498323] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e417637480 x1853164262358784/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2543.777347] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2545.229677] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 2545.234815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 2551.084883] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2553.142906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2562.273348] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 20:32:01 (1767317521) [ 2570.729691] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2572.907779] Lustre: Failing over lustre-MDT0000 [ 2573.149733] Lustre: server umount lustre-MDT0000 complete [ 2575.845888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2575.866379] LustreError: Skipped 1 previous similar message [ 2602.319744] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2602.324927] LDISKFS-fs (dm-0): recovery complete [ 2602.337946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2622.523924] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2623.573413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 2623.573722] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 2629.551633] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2631.231161] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2638.533457] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 20:33:18 (1767317598) [ 2646.523379] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2648.545148] Lustre: Failing over lustre-MDT0000 [ 2648.762557] Lustre: server umount lustre-MDT0000 complete [ 2677.126826] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2677.129962] LDISKFS-fs (dm-0): recovery complete [ 2677.136666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2694.586310] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2695.775910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 2695.784304] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:897) [ 2701.902904] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2703.179688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2711.516170] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 20:34:31 (1767317671) [ 2718.821531] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2720.507689] Lustre: Failing over lustre-MDT0000 [ 2720.848156] Lustre: server umount lustre-MDT0000 complete [ 2748.766361] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2748.769968] LDISKFS-fs (dm-0): recovery complete [ 2748.775946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.211666] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e4341adc00 x1853164262471040/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2762.569526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2762.575423] Lustre: Skipped 8 previous similar messages [ 2766.818169] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2768.032654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 2768.037562] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 2774.279747] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2775.863917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2784.234372] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 20:35:43 (1767317743) [ 2793.080750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2795.477490] Lustre: Failing over lustre-MDT0000 [ 2795.975942] Lustre: server umount lustre-MDT0000 complete [ 2826.489272] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2826.493068] LDISKFS-fs (dm-0): recovery complete [ 2826.500260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.179591] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2845.185844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2845.202806] Lustre: Skipped 36 previous similar messages [ 2845.367283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 2845.376322] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 2852.748882] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2854.593361] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2863.915669] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 20:37:03 (1767317823) [ 2871.078308] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2873.294297] Lustre: Failing over lustre-MDT0000 [ 2873.496538] Lustre: server umount lustre-MDT0000 complete [ 2902.311166] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2902.324255] LDISKFS-fs (dm-0): recovery complete [ 2902.354122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2916.842277] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e436d97100 x1853164262549120/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2916.865048] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 2921.616077] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2922.582722] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 2922.585425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:993) [ 2929.504740] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2931.148194] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2940.313995] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 20:38:20 (1767317900) [ 2948.665664] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2950.999903] Lustre: Failing over lustre-MDT0000 [ 2951.411664] Lustre: server umount lustre-MDT0000 complete [ 2953.194509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.225066] Lustre: Skipped 35 previous similar messages [ 2980.524465] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2980.527608] LDISKFS-fs (dm-0): recovery complete [ 2980.544815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.894522] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3000.017372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 3000.018035] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 3006.984691] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3009.039683] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3018.097070] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 20:39:37 (1767317977) [ 3026.089984] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3028.476328] Lustre: Failing over lustre-MDT0000 [ 3028.659911] LustreError: 87671:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3028.666176] LustreError: 87671:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 3029.023801] Lustre: server umount lustre-MDT0000 complete [ 3057.081983] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3057.087457] LDISKFS-fs (dm-0): recovery complete [ 3057.094770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3071.675266] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3071.696686] Lustre: Skipped 8 previous similar messages [ 3072.834399] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3072.840461] Lustre: Skipped 8 previous similar messages [ 3075.957569] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3077.232515] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3077.239261] Lustre: Skipped 8 previous similar messages [ 3077.268554] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 3077.270248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 3083.946605] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3085.745041] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3095.107839] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:40:54 (1767318054) [ 3103.224264] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3105.625194] Lustre: Failing over lustre-MDT0000 [ 3106.130747] Lustre: server umount lustre-MDT0000 complete [ 3118.048486] LustreError: 14434:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.066507] LustreError: 14434:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 374 previous similar messages [ 3124.191242] Lustre: 6800:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318069/real 1767318069] req@ffff92e305230000 x1853164262654080/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767318085 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3124.233604] Lustre: 6800:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 3124.240164] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3124.252781] LustreError: Skipped 8 previous similar messages [ 3134.885170] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3134.887811] LDISKFS-fs (dm-0): recovery complete [ 3134.893241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.908331] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3154.537223] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 3154.546991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 3161.388431] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3163.174814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3172.572473] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 20:42:12 (1767318132) [ 3176.953765] Lustre: 91556:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c549bbb0-37cc-46d8-8645-ca0ebda68de8 at adminstrative request [ 3184.772681] Lustre: Failing over lustre-MDT0000 [ 3185.082233] Lustre: server umount lustre-MDT0000 complete [ 3185.125031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3185.136736] LustreError: Skipped 1 previous similar message [ 3209.941050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3210.730203] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c35b1c [ 3215.431779] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3216.408363] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 3216.408473] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 3223.198459] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.814024] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3230.728968] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3239.369806] Lustre: DEBUG MARKER: before 3156, after 3156 [ 3247.086256] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 20:43:26 (1767318206) [ 3248.045410] Lustre: 93696:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c549bbb0-37cc-46d8-8645-ca0ebda68de8 at adminstrative request [ 3259.008821] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:43:38 (1767318218) [ 3269.406983] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3272.324089] Lustre: Failing over lustre-MDT0000 [ 3273.231373] Lustre: server umount lustre-MDT0000 complete [ 3305.381673] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3305.390539] LDISKFS-fs (dm-0): recovery complete [ 3305.404707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.809585] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c360a2 [ 3325.580892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 3325.596784] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 3325.618082] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3333.990521] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3335.650826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3346.356933] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:45:05 (1767318305) [ 3356.646442] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3359.571687] Lustre: Failing over lustre-MDT0000 [ 3360.144475] Lustre: server umount lustre-MDT0000 complete [ 3389.774489] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3389.777843] LDISKFS-fs (dm-0): recovery complete [ 3389.789577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3403.638449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3403.645721] Lustre: Skipped 7 previous similar messages [ 3408.999222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 3409.001155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 3409.160057] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3417.886922] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3419.805838] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3429.046986] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 20:46:28 (1767318388) [ 3436.465771] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3438.457337] Lustre: Failing over lustre-MDT0000 [ 3438.693742] Lustre: server umount lustre-MDT0000 complete [ 3467.357216] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3467.360446] LDISKFS-fs (dm-0): recovery complete [ 3467.373153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.548768] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e433d5b800 x1853164262835072/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3486.184501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3486.200516] Lustre: Skipped 33 previous similar messages [ 3486.271346] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3486.343516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 3486.344562] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 3494.849901] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3496.803868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3506.410933] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 20:47:45 (1767318465) [ 3517.515686] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3519.741793] Lustre: Failing over lustre-MDT0000 [ 3520.192690] Lustre: server umount lustre-MDT0000 complete [ 3549.167983] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3549.170831] LDISKFS-fs (dm-0): recovery complete [ 3549.178331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3567.854977] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3568.715402] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 3568.716859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 3575.896203] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3577.334374] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3586.019260] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 20:49:05 (1767318545) [ 3594.081787] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3596.160042] Lustre: Failing over lustre-MDT0000 [ 3596.439020] Lustre: server umount lustre-MDT0000 complete [ 3599.330678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.347385] Lustre: Skipped 32 previous similar messages [ 3625.153390] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3625.164725] LDISKFS-fs (dm-0): recovery complete [ 3625.188706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.054624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 3646.055409] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 3646.427374] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3656.899158] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3659.585240] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3669.611602] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:50:29 (1767318629) [ 3677.946879] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3680.753160] Lustre: Failing over lustre-MDT0000 [ 3681.079316] LustreError: 104982:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3681.092764] LustreError: 104982:0:(obd_class.h:479:obd_check_dev()) Skipped 63 previous similar messages [ 3681.332865] Lustre: server umount lustre-MDT0000 complete [ 3712.100351] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3712.102575] LDISKFS-fs (dm-0): recovery complete [ 3712.112302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.626830] LustreError: 19688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3718.647849] LustreError: 19688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 360 previous similar messages [ 3722.959426] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3722.961658] Lustre: Skipped 7 previous similar messages [ 3724.106623] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3724.120369] Lustre: Skipped 7 previous similar messages [ 3727.997740] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3728.440099] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3728.449460] Lustre: Skipped 7 previous similar messages [ 3728.485122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 3728.485268] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 3736.497960] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3738.653512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3747.646247] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:51:47 (1767318707) [ 3756.396683] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3758.777300] Lustre: Failing over lustre-MDT0000 [ 3759.072593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3759.081834] LustreError: Skipped 3 previous similar messages [ 3759.190855] Lustre: server umount lustre-MDT0000 complete [ 3774.432543] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318720/real 1767318720] req@ffff92e436d97480 x1853164262982912/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767318736 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3774.475515] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 3774.488712] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3774.504312] LustreError: Skipped 7 previous similar messages [ 3789.225259] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3789.227734] LDISKFS-fs (dm-0): recovery complete [ 3789.234654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.328993] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3806.822170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 3806.823322] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 3815.118718] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3816.923772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3826.096705] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 20:53:05 (1767318785) [ 3834.148206] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3836.128875] Lustre: Failing over lustre-MDT0000 [ 3836.471861] Lustre: server umount lustre-MDT0000 complete [ 3868.523964] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3868.529520] LDISKFS-fs (dm-0): recovery complete [ 3868.545084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3878.437281] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c387a7 [ 3883.825088] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3884.221964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 3884.222581] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 3892.026967] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3894.180471] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3905.111709] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:54:24 (1767318864) [ 3913.453523] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3915.952339] Lustre: Failing over lustre-MDT0000 [ 3916.205677] Lustre: server umount lustre-MDT0000 complete [ 3945.946677] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3945.949414] LDISKFS-fs (dm-0): recovery complete [ 3945.962436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3960.817695] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c38d81 [ 3966.569929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 3966.574263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 3966.915186] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3975.611336] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3977.628277] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3987.298324] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 20:55:47 (1767318947) [ 3994.953243] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3996.861367] Lustre: Failing over lustre-MDT0000 [ 3997.100693] Lustre: server umount lustre-MDT0000 complete [ 4024.786269] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4024.789447] LDISKFS-fs (dm-0): recovery complete [ 4024.796388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.846178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4037.849315] Lustre: Skipped 7 previous similar messages [ 4042.805778] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4043.348894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 4043.350569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 4051.777658] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4054.060855] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4065.199537] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 20:57:05 (1767319025) [ 4075.192723] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4076.993528] Lustre: Failing over lustre-MDT0000 [ 4077.482635] Lustre: server umount lustre-MDT0000 complete [ 4106.467436] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4106.470772] LDISKFS-fs (dm-0): recovery complete [ 4106.482723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.055808] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e4338a8000 x1853164263164288/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4121.092533] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 51 previous similar messages [ 4126.716953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4126.730248] Lustre: Skipped 33 previous similar messages [ 4126.734962] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4126.784076] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 4126.787229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 4135.106605] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4137.019356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4146.048138] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 20:58:25 (1767319105) [ 4146.924493] Lustre: 117245:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c549bbb0-37cc-46d8-8645-ca0ebda68de8 at adminstrative request [ 4155.949571] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 20:58:35 (1767319115) [ 4163.058199] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4165.200710] Lustre: Failing over lustre-MDT0000 [ 4165.558750] Lustre: server umount lustre-MDT0000 complete [ 4183.068462] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4183.073477] LDISKFS-fs (dm-0): recovery complete [ 4183.080850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4183.621701] Lustre: lustre-MDT0000: Aborting client recovery [ 4183.626353] LustreError: 118674:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4183.634768] Lustre: 118708:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4183.641741] Lustre: 118708:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c549bbb0-37cc-46d8-8645-ca0ebda68de8@ [ 4183.656207] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4183.666442] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 4183.682954] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 4183.786397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1479 to 0x280000401:1505) [ 4183.788443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1480 to 0x2c0000401:1505) [ 4188.641156] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4189.154759] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4213.046298] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 20:59:32 (1767319172) [ 4221.105860] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4223.306304] Lustre: Failing over lustre-MDT0000 [ 4223.676513] Lustre: server umount lustre-MDT0000 complete [ 4224.479986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4224.496534] Lustre: Skipped 28 previous similar messages [ 4239.880343] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4239.882621] LDISKFS-fs (dm-0): recovery complete [ 4239.890631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4240.318380] Lustre: *** cfs_fail_loc=1311, val=0*** [ 4240.363461] Lustre: lustre-MDT0000: Aborting client recovery [ 4240.369468] LustreError: 120667:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4240.392850] Lustre: 120700:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4240.409488] Lustre: 120700:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 4240.424566] Lustre: 120700:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4240.434813] Lustre: 120700:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4240.452458] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4240.475040] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 4240.490233] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 4240.562432] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 4240.566831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 4244.966723] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4245.497755] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4259.125542] Lustre: *** cfs_fail_loc=1311, val=0*** [ 4259.130264] Lustre: Skipped 1 previous similar message [ 4267.050347] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 21:00:26 (1767319226) [ 4274.890179] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4276.893752] Lustre: Failing over lustre-MDT0000 [ 4277.197177] Lustre: server umount lustre-MDT0000 complete [ 4293.640187] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4293.642260] LDISKFS-fs (dm-0): recovery complete [ 4293.656762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4294.108454] Lustre: lustre-MDT0000: Aborting client recovery [ 4294.110836] LustreError: 122665:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4294.116324] Lustre: 122698:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4294.125453] Lustre: 122698:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 4294.138234] Lustre: 122698:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4294.142276] Lustre: 122698:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4294.146588] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4294.159477] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 4294.180260] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 4294.260853] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 4294.264068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 4298.503847] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4299.251879] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4320.224442] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 21:01:19 (1767319279) [ 4321.439187] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4321.442351] LustreError: 19688:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e417321f80 x1853164181778944/t201863462916(0) o36->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:194/0 lens 512/456 e 0 to 0 dl 1767319294 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 4325.218139] Lustre: Failing over lustre-MDT0000 [ 4325.338725] LustreError: 123535:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4325.342968] LustreError: 123535:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 4325.416826] Lustre: server umount lustre-MDT0000 complete [ 4327.261597] LustreError: 14435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4327.278507] LustreError: 14435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 302 previous similar messages [ 4336.439883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4336.730037] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4336.730355] Lustre: lustre-MDT0000: Aborting client recovery [ 4336.737328] Lustre: Skipped 14 previous similar messages [ 4336.744812] LustreError: 124130:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4336.749844] Lustre: 124163:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4336.753093] Lustre: 124163:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 4336.757379] Lustre: 124163:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4336.765736] Lustre: 124163:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4336.771833] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4336.787191] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 4336.800861] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 4336.858858] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1601) [ 4336.865059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 4341.202582] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4341.743052] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4363.233454] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 4365.243207] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 21:02:04 (1767319324) [ 4372.613618] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4375.411300] Lustre: Failing over lustre-MDT0000 [ 4375.893559] Lustre: server umount lustre-MDT0000 complete [ 4377.568583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4377.575075] LustreError: Skipped 5 previous similar messages [ 4390.889520] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4390.893666] LDISKFS-fs (dm-0): recovery complete [ 4390.903352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4391.094739] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4391.104800] LustreError: Skipped 8 previous similar messages [ 4391.468960] Lustre: lustre-MDT0000: Aborting client recovery [ 4391.472367] LustreError: 126221:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 4391.480803] Lustre: 126255:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 4391.485568] Lustre: 126255:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 4391.491164] Lustre: 126255:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 4391.497226] Lustre: 126255:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 4391.501713] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 4391.507285] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 4391.519229] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 4391.562461] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1633) [ 4391.564502] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 4395.562830] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4396.546957] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 4419.228549] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 21:02:58 (1767319378) [ 4455.230399] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4457.158532] Lustre: Failing over lustre-MDT0000 [ 4457.579167] Lustre: server umount lustre-MDT0000 complete [ 4478.927191] Lustre: 6799:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767319424/real 1767319424] req@ffff92e42b911500 x1853164263437824/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767319440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4478.991238] Lustre: 6799:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4488.438699] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4488.441501] LDISKFS-fs (dm-0): recovery complete [ 4488.449695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4492.108929] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4492.121228] Lustre: Skipped 5 previous similar messages [ 4494.449600] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4494.983429] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4494.998261] Lustre: Skipped 5 previous similar messages [ 4495.061948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 4495.063625] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 4503.227384] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4505.293371] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4527.801913] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 21:04:47 (1767319487) [ 4557.774807] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4568.982590] Lustre: Failing over lustre-MDT0000 [ 4569.324833] Lustre: server umount lustre-MDT0000 complete [ 4599.303575] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4599.306305] LDISKFS-fs (dm-0): recovery complete [ 4599.314413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4619.545874] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4623.983784] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 4623.983988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 4627.961393] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4630.006087] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4650.972509] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 21:06:50 (1767319610) [ 4653.155150] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 4654.085569] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4654.091982] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4661.222262] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 21:07:01 (1767319621) [ 4691.847652] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4703.289829] Lustre: Failing over lustre-OST0000 [ 4703.461582] Lustre: server umount lustre-OST0000 complete [ 4734.690478] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 4734.696756] LDISKFS-fs (dm-2): recovery complete [ 4734.715609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4735.104962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4735.122822] Lustre: Skipped 8 previous similar messages [ 4737.363817] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4737.368084] Lustre: Skipped 32 previous similar messages [ 4742.545904] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4802.767614] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 21:09:22 (1767319762) [ 4811.496962] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4814.743319] Lustre: Failing over lustre-MDT0000 [ 4815.088294] Lustre: server umount lustre-MDT0000 complete [ 4846.555831] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4846.558348] LDISKFS-fs (dm-0): recovery complete [ 4846.568611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.564583] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4864.630665] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 4864.632588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 4872.464549] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4874.738941] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4880.863837] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4880.884187] Lustre: Skipped 30 previous similar messages [ 4880.887100] LustreError: 136433:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 4880.888088] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 4881.951581] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 4894.296704] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 21:10:53 (1767319853) [ 4900.151919] LustreError: 19688:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4905.439640] LustreError: 19688:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4905.448448] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnecting [ 4905.514282] LustreError: 57337:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 waking [ 4907.537986] LustreError: 14436:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4912.607848] LustreError: 14436:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4912.613668] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnecting [ 4914.842982] LustreError: 14436:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4920.287345] LustreError: 14436:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4920.300909] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnecting [ 4922.656478] LustreError: 19688:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4927.968262] LustreError: 19688:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4930.341283] LustreError: 14435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4935.649614] LustreError: 14435:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4935.655186] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnecting [ 4935.661982] Lustre: Skipped 1 previous similar message [ 4945.161376] LustreError: 19688:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4945.169223] LustreError: 19688:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4950.495238] LustreError: 19688:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4950.507925] LustreError: 19688:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 4957.663201] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnecting [ 4957.673572] Lustre: Skipped 2 previous similar messages [ 4967.999468] LustreError: 14436:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 4968.022227] LustreError: 14436:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4973.538386] LustreError: 14436:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 4973.552444] LustreError: 14436:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 4984.189429] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 21:12:23 (1767319943) [ 4986.817558] LustreError: 14436:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 4996.937949] Lustre: lustre-MDT0000: Export ffff92e4300ce800 already connecting from 192.168.203.23@tcp [ 5002.048479] Lustre: lustre-MDT0000: Export ffff92e4300ce800 already connecting from 192.168.203.23@tcp [ 5007.169232] Lustre: lustre-MDT0000: Export ffff92e4300ce800 already connecting from 192.168.203.23@tcp [ 5009.778392] Lustre: lustre-MDT0000: Export ffff92e4300ce800 already connecting from 192.168.203.23@tcp [ 5017.421794] Lustre: lustre-MDT0000: Export ffff92e4300ce800 already connecting from 192.168.203.23@tcp [ 5017.440597] Lustre: Skipped 1 previous similar message [ 5026.895960] LustreError: 14436:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 5026.907957] Lustre: 14436:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff92e4333d2d80 x1853164184442880/t0(0) o38->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:0/0 lens 520/416 e 0 to 0 dl 1767319968 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 5027.653667] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnecting [ 5027.663316] Lustre: Skipped 3 previous similar messages [ 5027.678654] LustreError: 19688:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 5052.226824] Lustre: lustre-MDT0000: Export ffff92e4300ce800 already connecting from 192.168.203.23@tcp [ 5052.234929] Lustre: Skipped 1 previous similar message [ 5067.687199] LustreError: 19688:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 5067.695895] Lustre: 19688:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff92e42b69fb80 x1853164184446848/t0(0) o38->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:0/0 lens 520/416 e 0 to 0 dl 1767320009 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5072.713708] LustreError: 14435:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 5097.287141] Lustre: lustre-MDT0000: Export ffff92e4300ce800 already connecting from 192.168.203.23@tcp [ 5097.296531] Lustre: Skipped 4 previous similar messages [ 5112.767844] LustreError: 14435:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 5112.785230] Lustre: 14435:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff92e30690f100 x1853164184450432/t0(0) o38->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:0/0 lens 520/416 e 0 to 0 dl 1767320054 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5117.762199] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnecting [ 5117.770888] Lustre: Skipped 1 previous similar message [ 5117.774579] LustreError: 14436:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 5143.384740] Lustre: lustre-MDT0000: Export ffff92e4300ce800 already connecting from 192.168.203.23@tcp [ 5143.389162] Lustre: Skipped 3 previous similar messages [ 5157.831530] LustreError: 14436:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 5157.842386] Lustre: 14436:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff92e436f3d500 x1853164184453888/t0(0) o38->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:0/0 lens 520/416 e 0 to 0 dl 1767320099 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5158.720993] LustreError: 14435:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 5198.767258] LustreError: 14435:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 5198.778263] Lustre: 14435:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff92e41778e680 x1853164184457088/t0(0) o38->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:0/0 lens 520/416 e 0 to 0 dl 1767320140 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5203.800629] LustreError: 23251:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 5217.272359] LustreError: 23251:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 5222.738982] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 21:16:22 (1767320182) [ 5231.146086] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5235.637358] Lustre: Failing over lustre-MDT0000 [ 5235.863443] LustreError: 140543:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5235.871163] LustreError: 140543:0:(obd_class.h:479:obd_check_dev()) Skipped 43 previous similar messages [ 5236.020757] Lustre: server umount lustre-MDT0000 complete [ 5238.242932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5238.254973] LustreError: Skipped 6 previous similar messages [ 5238.261435] LustreError: 14440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.282287] LustreError: 14440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 188 previous similar messages [ 5253.894386] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5253.897661] LDISKFS-fs (dm-0): recovery complete [ 5253.906354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.113710] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5254.122119] LustreError: Skipped 3 previous similar messages [ 5254.296905] Lustre: *** cfs_fail_loc=712, val=0*** [ 5254.303610] LustreError: 17336:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff92e4338c5f80 x1853164264172800/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 5254.406073] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5254.407445] Lustre: lustre-MDT0000: Aborting client recovery [ 5254.411412] Lustre: Skipped 9 previous similar messages [ 5254.418474] LustreError: 141333:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5254.424502] Lustre: 141366:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5254.429736] Lustre: 141366:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 5254.435850] Lustre: 141366:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 5254.447499] Lustre: 141366:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 5254.455497] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 5254.464075] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 5254.477412] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 5254.534225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 5254.535483] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 5259.338344] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5259.751425] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5276.853357] Lustre: Failing over lustre-MDT0000 [ 5277.382936] Lustre: server umount lustre-MDT0000 complete [ 5296.080147] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320241/real 1767320241] req@ffff92e433a3aa00 x1853164264191616/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767320257 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5296.127866] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5303.771937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.951124] LustreError: 142661:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 5305.962234] LustreError: 142661:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff92e302535180 x1853164264197504/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767320267 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5305.986883] LustreError: 142661:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 5308.256749] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5308.289144] Lustre: Skipped 3 previous similar messages [ 5310.918836] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5312.000685] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5312.013904] Lustre: Skipped 3 previous similar messages [ 5312.085580] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 5312.090453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 5320.013595] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5321.940293] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5331.546395] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 21:18:11 (1767320291) [ 5331.650606] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnecting [ 5331.660258] Lustre: Skipped 2 previous similar messages [ 5340.867243] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 21:18:20 (1767320300) [ 5342.032247] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 5342.039555] LustreError: 14443:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e4177d5180 x1853164184521728/t0(0) o700->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:459/0 lens 264/248 e 0 to 0 dl 1767320314 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 5363.503258] Lustre: Failing over lustre-MDT0000 [ 5363.534821] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 5363.802621] Lustre: server umount lustre-MDT0000 complete [ 5389.317744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5394.208603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5394.212617] Lustre: Skipped 3 previous similar messages [ 5399.030248] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5399.522074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5399.533284] Lustre: Skipped 14 previous similar messages [ 5399.615209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 5399.619120] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 5407.433962] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5409.642673] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5423.577865] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 21:19:43 (1767320383) [ 5426.119381] Lustre: Failing over lustre-OST0000 [ 5426.297629] Lustre: server umount lustre-OST0000 complete [ 5449.592458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5456.921728] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5464.948821] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5467.338680] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5542.777862] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 21:21:41 (1767320501) [ 5551.006936] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5553.858607] Lustre: Failing over lustre-MDT0000 [ 5554.110074] Lustre: server umount lustre-MDT0000 complete [ 5557.225042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5557.237774] Lustre: Skipped 16 previous similar messages [ 5582.886789] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5582.891343] LDISKFS-fs (dm-0): recovery complete [ 5582.901578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5598.208771] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e43fb3fb80 x1853164264352256/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5598.251174] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 11 previous similar messages [ 5604.247531] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5604.625376] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 5604.628995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 5679.458253] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 21:23:59 (1767320639) [ 5681.703472] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5681.706976] Lustre: Skipped 2 previous similar messages [ 5694.486977] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 21:24:14 (1767320654) [ 5697.652688] Lustre: Failing over lustre-MDT0000 [ 5697.914325] Lustre: server umount lustre-MDT0000 complete [ 5721.332249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5732.211457] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5733.373210] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5733.380473] LustreError: 150029:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e41711ad80 x1853164184657152/t0(0) o101->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:134/0 lens 328/344 e 0 to 0 dl 1767320744 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 5788.483537] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [ 5788.518921] Lustre: 150029:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 5788.633884] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 5788.640434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 5793.324883] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5795.106476] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5804.030074] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 21:26:03 (1767320763) [ 5806.239370] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5814.758644] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5816.570602] Lustre: Failing over lustre-MDT0000 [ 5816.756804] Lustre: server umount lustre-MDT0000 complete [ 5840.865231] LustreError: 16316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5840.888608] LustreError: 16316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 197 previous similar messages [ 5844.371291] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5844.373153] LDISKFS-fs (dm-0): recovery complete [ 5844.379615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.471126] LustreError: 152284:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 5845.484437] LustreError: 152284:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff92e42b828e00 x1853164264477184/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767320807 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5845.507385] LustreError: 152284:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 5845.996315] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c826d8 [ 5850.660940] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5851.705253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3137) [ 5851.705653] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3105) [ 5858.448693] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5860.327837] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5869.923952] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 21:27:09 (1767320829) [ 5871.787491] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5882.843915] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5885.064325] Lustre: Failing over lustre-MDT0000 [ 5885.253039] LustreError: 153589:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5885.272402] LustreError: 153589:0:(obd_class.h:479:obd_check_dev()) Skipped 51 previous similar messages [ 5885.478427] Lustre: server umount lustre-MDT0000 complete [ 5904.356538] Lustre: 6799:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767320849/real 1767320849] req@ffff92e41713b800 x1853164264507776/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767320865 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5904.379206] Lustre: 6799:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5904.387872] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5904.404410] LustreError: Skipped 5 previous similar messages [ 5912.516338] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5912.524945] LDISKFS-fs (dm-0): recovery complete [ 5912.537125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5914.592664] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e43419b480 x1853164264515712/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5914.859923] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5914.864400] Lustre: Skipped 8 previous similar messages [ 5916.076534] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5916.088112] Lustre: Skipped 5 previous similar messages [ 5919.685939] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5920.288917] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5920.303659] Lustre: Skipped 5 previous similar messages [ 5920.367626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3169) [ 5920.369554] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 5927.963956] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5929.861992] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5939.634994] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 21:28:19 (1767320899) [ 5941.216893] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 5943.246933] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 5950.628529] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5952.530894] Lustre: Failing over lustre-MDT0000 [ 5952.943232] Lustre: server umount lustre-MDT0000 complete [ 5956.065179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5956.075615] LustreError: Skipped 3 previous similar messages [ 5979.188854] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5979.191403] LDISKFS-fs (dm-0): recovery complete [ 5979.196435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5981.669919] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c8335e [ 5986.484192] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5987.358481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3201) [ 5987.363788] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 5997.954867] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 21:29:17 (1767320957) [ 6000.400507] Lustre: *** cfs_fail_loc=13b, val=315*** [ 6000.402519] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 6000.407133] LustreError: 14437:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e436c57100 x1853164184720896/t257698037777(0) o35->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:362/0 lens 392/456 e 0 to 0 dl 1767320972 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 6003.279710] Lustre: Failing over lustre-MDT0000 [ 6003.505990] Lustre: server umount lustre-MDT0000 complete [ 6027.976844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6034.686818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.694636] Lustre: Skipped 6 previous similar messages [ 6039.464681] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6040.052218] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6040.066761] Lustre: Skipped 29 previous similar messages [ 6040.143944] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3201) [ 6040.146023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 6040.159888] Lustre: 14437:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e433af0a80 x1853164184720896/t257698037777(0) o35->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:402/0 lens 392/456 e 0 to 0 dl 1767321012 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 6047.944645] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6049.804839] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6058.650662] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 21:30:18 (1767321018) [ 6060.123575] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6060.133864] LustreError: 19688:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e42b82aa00 x1853164184736640/t261993005072(0) o36->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:466/0 lens 504/448 e 0 to 0 dl 1767321076 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 6070.275543] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6072.231172] Lustre: Failing over lustre-MDT0000 [ 6072.777069] Lustre: server umount lustre-MDT0000 complete [ 6100.546053] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6100.556937] LDISKFS-fs (dm-0): recovery complete [ 6100.566739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6106.752936] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6107.247837] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3265) [ 6107.257897] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 6107.274695] Lustre: 70114:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e4338a9880 x1853164184736640/t261993005072(0) o36->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:513/0 lens 504/2880 e 0 to 0 dl 1767321123 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 6115.089257] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6117.271868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6125.834180] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 21:31:26 (1767321086) [ 6127.176557] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6127.182667] LustreError: 19688:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e43315b800 x1853164184753664/t266287972368(0) o36->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:533/0 lens 504/448 e 0 to 0 dl 1767321143 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 6129.394666] Lustre: *** cfs_fail_loc=13b, val=315*** [ 6137.199869] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6139.346310] Lustre: Failing over lustre-MDT0000 [ 6139.689084] Lustre: server umount lustre-MDT0000 complete [ 6167.845225] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6167.856704] LDISKFS-fs (dm-0): recovery complete [ 6167.865305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6168.546880] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c84674 [ 6173.156886] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6174.300603] Lustre: 14437:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e417a73450 x1853164184754560/t266287972369(0) o35->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:536/0 lens 392/456 e 0 to 0 dl 1767321146 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 6174.310982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3297) [ 6174.328387] Lustre: 14437:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6174.337283] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 6185.303364] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 21:32:24 (1767321144) [ 6186.794965] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6186.802801] Lustre: Skipped 1 previous similar message [ 6186.806558] LustreError: 23251:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e409ee4000 x1853164184768896/t270582939664(0) o36->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:593/0 lens 504/448 e 0 to 0 dl 1767321203 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 6186.840558] LustreError: 23251:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 6188.648191] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 6196.356965] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6198.033142] Lustre: Failing over lustre-MDT0000 [ 6198.317111] Lustre: server umount lustre-MDT0000 complete [ 6199.775583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6199.784925] Lustre: Skipped 31 previous similar messages [ 6227.712232] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6227.717947] LDISKFS-fs (dm-0): recovery complete [ 6227.730898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.760019] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e417323480 x1853164264693632/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6241.803095] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 14 previous similar messages [ 6246.931573] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6247.474276] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3297) [ 6247.476033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 6247.504830] Lustre: 14434:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e436c54700 x1853164184768896/t270582939664(0) o36->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:654/0 lens 504/2880 e 0 to 0 dl 1767321264 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 6258.008347] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 21:33:37 (1767321217) [ 6259.281568] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 6260.955983] Lustre: *** cfs_fail_loc=13b, val=315*** [ 6260.957764] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 6260.962612] LustreError: 14437:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e417a72c50 x1853164184785024/t274877906960(0) o35->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:623/0 lens 392/456 e 0 to 0 dl 1767321233 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 6269.279758] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6271.179162] Lustre: Failing over lustre-MDT0000 [ 6271.381773] Lustre: server umount lustre-MDT0000 complete [ 6301.279470] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6301.282430] LDISKFS-fs (dm-0): recovery complete [ 6301.290475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6313.973654] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99c85308 [ 6319.048650] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6319.662146] Lustre: 14437:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e433853100 x1853164184785024/t274877906960(0) o35->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:682/0 lens 392/456 e 0 to 0 dl 1767321292 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 6319.669183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 6319.670137] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3329) [ 6330.970923] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 21:34:50 (1767321290) [ 6331.981775] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 6331.991694] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 6331.998727] LustreError: 14436:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e4331d6a00 x1853164184798464/t279172874255(0) o101->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:738/0 lens 664/608 e 0 to 0 dl 1767321348 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 6392.639556] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnecting [ 6392.648973] Lustre: Skipped 1 previous similar message [ 6392.662889] Lustre: 14434:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e433b51500 x1853164184798464/t279172874255(0) o101->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:44/0 lens 664/3488 e 0 to 0 dl 1767321409 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 6399.532609] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 21:35:59 (1767321359) [ 6406.085960] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6407.578483] Lustre: Failing over lustre-MDT0000 [ 6407.804385] Lustre: server umount lustre-MDT0000 complete [ 6435.710388] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 6435.713391] LDISKFS-fs (dm-0): recovery complete [ 6435.733249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.579965] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6443.092834] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 6443.093549] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3361) [ 6449.833444] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6451.594065] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6471.834736] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 21:37:11 (1767321431) [ 6481.667093] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6483.496640] Lustre: Failing over lustre-MDT0000 [ 6483.826368] Lustre: server umount lustre-MDT0000 complete [ 6483.941425] LustreError: 16316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6483.976609] LustreError: 16316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 311 previous similar messages [ 6504.415882] Lustre: 6797:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767321450/real 1767321450] req@ffff92e4331d4700 x1853164264823552/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767321466 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6504.452072] Lustre: 6797:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 6504.456527] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6504.461686] LustreError: Skipped 7 previous similar messages [ 6512.409561] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6512.412697] LDISKFS-fs (dm-0): recovery complete [ 6512.419583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.527117] LustreError: 171419:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6514.536261] LustreError: 171419:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff92e433af3480 x1853164264828928/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767321476 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6514.558387] LustreError: 171419:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6514.943887] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6514.945839] Lustre: Skipped 7 previous similar messages [ 6516.546127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6516.552757] Lustre: Skipped 7 previous similar messages [ 6519.038615] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6520.401582] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6520.412632] Lustre: Skipped 7 previous similar messages [ 6520.483351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 6520.488278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3425) [ 6527.308966] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6529.379803] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6535.743870] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 6548.510810] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 21:38:28 (1767321508) [ 6590.065807] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6592.218349] Lustre: Failing over lustre-MDT0000 [ 6592.540194] LustreError: 173170:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6592.550147] LustreError: 173170:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 6592.664807] Lustre: server umount lustre-MDT0000 complete [ 6597.088093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6597.099508] LustreError: Skipped 5 previous similar messages [ 6620.734178] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6620.736263] LDISKFS-fs (dm-0): recovery complete [ 6620.741441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6626.327960] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6628.696178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 6628.701139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 6633.825863] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6635.566187] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6688.228416] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 21:40:47 (1767321647) [ 6700.042010] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6702.252756] Lustre: Failing over lustre-MDT0000 [ 6702.617341] Lustre: server umount lustre-MDT0000 complete [ 6730.967807] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6730.970115] LDISKFS-fs (dm-0): recovery complete [ 6730.980607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.081146] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e43401fb80 x1853164265593344/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6746.104502] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 7 previous similar messages [ 6746.377085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6746.387334] Lustre: Skipped 7 previous similar messages [ 6750.733776] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6751.731246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6751.740302] Lustre: Skipped 33 previous similar messages [ 6751.865616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 6751.871370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 6757.971458] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6759.941072] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6770.239575] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 6771.640743] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 6778.381449] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 21:42:18 (1767321738) [ 6787.377989] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 6850.284321] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6850.296647] LustreError: 14436:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e433159500 x1853164187609216/t296352743435(0) o36->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:501/0 lens 66040/440 e 0 to 0 dl 1767321866 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 6909.786757] Lustre: 14436:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e417a68450 x1853164187609216/t296352743435(0) o36->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:561/0 lens 66040/440 e 0 to 0 dl 1767321926 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 6919.259822] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 6920.831601] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 21:44:40 (1767321880) [ 6934.170278] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6936.430934] Lustre: Failing over lustre-MDT0000 [ 6940.498727] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 6941.152331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.155479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6941.169662] Lustre: Skipped 26 previous similar messages [ 6941.179399] Lustre: Skipped 2 previous similar messages [ 6942.256576] Lustre: server umount lustre-MDT0000 complete [ 6970.809811] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6970.815083] LDISKFS-fs (dm-0): recovery complete [ 6970.824640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6976.549834] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6978.772436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 6978.774305] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 6984.736729] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6986.508862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6997.009780] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 21:45:56 (1767321956) [ 7021.178483] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7033.994589] Lustre: Failing over lustre-OST0000 [ 7034.163453] Lustre: server umount lustre-OST0000 complete [ 7059.727446] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7059.730219] LDISKFS-fs (dm-2): recovery complete [ 7059.738444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7066.482904] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7080.527648] Lustre: Failing over lustre-OST0000 [ 7080.614496] Lustre: server umount lustre-OST0000 complete [ 7085.536303] LustreError: 27890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7085.567418] LustreError: 27890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 159 previous similar messages [ 7101.399602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7108.153263] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7116.607436] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7118.420760] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7159.163900] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 21:48:38 (1767322118) [ 7162.068550] Lustre: Failing over lustre-MDT0000 [ 7162.397025] Lustre: server umount lustre-MDT0000 complete [ 7178.719792] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767322124/real 1767322124] req@ffff92e433b50a80 x1853164265944192/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767322140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7178.761533] Lustre: 6798:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 7178.776530] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7178.784928] LustreError: Skipped 3 previous similar messages [ 7186.439606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7188.639778] LustreError: 184864:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7188.653552] LustreError: 184864:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff92e30267e680 x1853164265950080/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767322150 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7188.678588] LustreError: 184864:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7189.259404] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7189.269966] Lustre: Skipped 5 previous similar messages [ 7189.565771] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7189.571639] Lustre: Skipped 5 previous similar messages [ 7193.125678] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7194.617197] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7194.636336] Lustre: Skipped 5 previous similar messages [ 7194.700347] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 7194.700589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 7207.803714] Lustre: Failing over lustre-MDT0000 [ 7207.944295] LustreError: 185357:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7207.949245] LustreError: 185357:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 7208.048064] Lustre: server umount lustre-MDT0000 complete [ 7209.952795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7209.961720] LustreError: Skipped 13 previous similar messages [ 7231.902751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7235.554845] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99cdabf8 [ 7239.651134] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7241.260844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 7241.261561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 7246.568214] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7248.097205] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7256.835637] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 21:50:16 (1767322216) [ 7271.820555] Lustre: Failing over lustre-OST0000 [ 7272.041765] Lustre: server umount lustre-OST0000 complete [ 7295.123969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7301.356507] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7308.414515] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7310.019908] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7319.295061] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 21:51:19 (1767322279) [ 7323.719995] Lustre: Failing over lustre-MDT0000 [ 7323.944701] Lustre: server umount lustre-MDT0000 complete [ 7344.622244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7344.708435] Lustre: *** cfs_fail_loc=605, val=0*** [ 7344.710846] LustreError: 189765:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc09caab0 failed: rc = -95 [ 7344.717550] LustreError: 189765:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 7344.722949] LustreError: 189765:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 7344.728152] LustreError: 189765:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 7344.732419] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 7344.737861] LustreError: 189765:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 7344.746915] Lustre: server umount lustre-MDT0000 complete [ 7344.750210] LustreError: 189765:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 7363.200030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.155774] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99cdb1a8 [ 7367.174218] Lustre: MGC192.168.203.123@tcp: Connection restored to 0@lo (at 0@lo) [ 7367.180689] Lustre: Skipped 22 previous similar messages [ 7367.509337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7367.518206] Lustre: Skipped 6 previous similar messages [ 7371.427191] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7372.866754] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 7372.870317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 7379.131612] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 21:52:19 (1767322339) [ 7385.870718] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7388.688432] Lustre: Failing over lustre-MDT0000 [ 7388.889303] Lustre: server umount lustre-MDT0000 complete [ 7418.398374] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7418.403201] LDISKFS-fs (dm-0): recovery complete [ 7418.412480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7421.808155] Lustre: *** cfs_fail_loc=707, val=0*** [ 7423.070275] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7482.214718] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 7482.283575] Lustre: 192889:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 7482.298783] Lustre: 192889:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 7482.710316] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 7482.710783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 7486.884519] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7488.192524] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7497.418387] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 21:54:17 (1767322457) [ 7528.700763] LustreError: 14436:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff92e433a77100 x1853164188513536/t0(0) o101->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:381/0 lens 664/0 e 0 to 0 dl 1767322501 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7528.731948] LustreError: 14436:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 7539.783154] LustreError: 14436:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7539.816372] LustreError: 14438:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff92e4331d6300 x1853164188514816/t0(0) o35->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:392/0 lens 392/0 e 0 to 0 dl 1767322512 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7541.787529] LustreError: 14436:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff92e417a55500 x1853164188520704/t0(0) o101->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:433/0 lens 576/0 e 0 to 0 dl 1767322553 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:0 [ 7541.812683] LustreError: 14436:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 27 previous similar messages [ 7545.828156] LustreError: 17605:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff92e4176db800 x1853164266142336/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:398/0 lens 544/0 e 0 to 0 dl 1767322518 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 7545.866791] LustreError: 17605:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 28 previous similar messages [ 7557.947825] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 21:55:17 (1767322517) [ 7587.872848] LustreError: 55998:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 7598.921400] LustreError: 55998:0:(tgt_handler.c:2782:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 7607.436794] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 21:56:07 (1767322567) [ 7635.663087] LustreError: 70114:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff92e409ee5500 x1853164188539648/t0(0) o101->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:488/0 lens 576/0 e 0 to 0 dl 1767322608 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7635.702196] LustreError: 70114:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 7640.711879] LustreError: 70114:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7646.698900] LustreError: 57343:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff92e4170bbb80 x1853164266191360/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:499/0 lens 544/0 e 0 to 0 dl 1767322619 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 7646.737828] LustreError: 57343:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 110 previous similar messages [ 7653.016206] LustreError: 14435:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7671.808798] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 21:57:11 (1767322631) [ 7774.223609] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 21:58:54 (1767322734) [ 7801.907175] LustreError: 14436:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff92e43453df80 x1853164188620032/t0(0) o101->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:654/0 lens 576/0 e 0 to 0 dl 1767322774 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 7801.922967] LustreError: 14436:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 108 previous similar messages [ 7801.929580] LustreError: 14436:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 7801.936389] LustreError: 14436:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 7802.359449] LustreError: 14436:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7818.148208] LustreError: 14438:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 7818.151761] LustreError: 14438:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 7818.575129] LustreError: 14438:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 7818.582552] LustreError: 14438:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 37 previous similar messages [ 7833.972529] LustreError: 14434:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff92e43fac6d80 x1853164188639104/t0(0) o36->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:686/0 lens 504/0 e 0 to 0 dl 1767322806 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 7833.996556] LustreError: 14434:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 78 previous similar messages [ 7850.079697] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 22:00:10 (1767322810) [ 7881.770715] Lustre: DEBUG MARKER: phase 2 [ 7890.684544] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 22:00:50 (1767322850) [ 7969.768041] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 22:02:09 (1767322929) [ 7971.142341] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 7972.611763] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 22:02:12 (1767322932) [ 7976.488542] Lustre: DEBUG MARKER: Started rundbench load pid=140984 ... [ 7983.596801] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7986.019503] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 7987.657555] Lustre: Failing over lustre-MDT0000 [ 7987.871515] LustreError: 199850:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7987.885185] LustreError: 199850:0:(obd_class.h:479:obd_check_dev()) Skipped 27 previous similar messages [ 7988.118359] Lustre: server umount lustre-MDT0000 complete [ 7989.086085] LustreError: 14436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7989.103750] LustreError: 14436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 160 previous similar messages [ 7989.737038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7989.746556] LustreError: Skipped 2 previous similar messages [ 7989.754134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.771441] Lustre: Skipped 22 previous similar messages [ 8006.498347] Lustre: 6799:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767322952/real 1767322952] req@ffff92e433850000 x1853164266429440/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767322968 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8006.523592] Lustre: 6799:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 8006.527736] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8006.535227] LustreError: Skipped 3 previous similar messages [ 8010.754590] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 8010.761913] LDISKFS-fs (dm-0): recovery complete [ 8010.773987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8016.872264] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e436c54700 x1853164266589696/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8016.887674] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 8017.050016] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.055723] Lustre: Skipped 1 previous similar message [ 8017.090431] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8017.094923] Lustre: Skipped 4 previous similar messages [ 8017.833447] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8017.841088] Lustre: Skipped 4 previous similar messages [ 8019.759846] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8022.504662] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8022.513379] Lustre: Skipped 8 previous similar messages [ 8022.569562] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 8022.574194] Lustre: Skipped 4 previous similar messages [ 8022.620474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 8022.621960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 8025.128702] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8026.414979] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8034.254626] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8036.497690] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 8037.901512] Lustre: Failing over lustre-MDT0001 [ 8037.913265] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 8037.921711] Lustre: Skipped 1 previous similar message [ 8038.071161] Lustre: server umount lustre-MDT0001 complete [ 8058.802500] LDISKFS-fs (dm-1): 9 truncates cleaned up [ 8058.803833] LDISKFS-fs (dm-1): recovery complete [ 8058.810708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8061.768249] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8064.488590] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:570 to 0x2c0000400:609) [ 8064.489817] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:589 to 0x280000400:609) [ 8067.854778] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8069.103777] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8076.670679] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8079.029465] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 8080.699225] Lustre: Failing over lustre-MDT0000 [ 8081.011798] Lustre: server umount lustre-MDT0000 complete [ 8103.203430] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 8103.205470] LDISKFS-fs (dm-0): recovery complete [ 8103.212759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8110.562646] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99d1049f [ 8113.672344] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8116.290516] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 8116.293188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 8119.721942] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8121.150996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8127.544037] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 22:04:47 (1767323087) [ 8255.632547] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8267.189362] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 8268.571119] Lustre: Failing over lustre-MDT0001 [ 8268.601091] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 8268.818804] Lustre: server umount lustre-MDT0001 complete [ 8288.948827] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8288.951432] LDISKFS-fs (dm-1): recovery complete [ 8288.959233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8291.546554] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8296.979203] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1095 to 0x2c0000400:1121) [ 8296.979445] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1096 to 0x280000400:1121) [ 8299.521102] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8300.710638] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8427.797562] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8439.276357] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 8440.636094] Lustre: Failing over lustre-MDT0001 [ 8440.653361] LustreError: 14421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323402 with bad export cookie 17006620084285081684 [ 8440.666245] LustreError: 14421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8441.051578] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 8443.712226] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 8443.716317] Lustre: Skipped 3 previous similar messages [ 8446.080822] Lustre: server umount lustre-MDT0001 complete [ 8465.711814] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8465.713768] LDISKFS-fs (dm-1): recovery complete [ 8465.721816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8468.606479] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8473.386250] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1482 to 0x280000400:1505) [ 8473.388306] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1481 to 0x2c0000400:1505) [ 8475.856345] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8477.029577] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8520.563731] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 22:11:20 (1767323480) [ 8646.182093] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 8657.600420] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 8658.649751] Lustre: Failing over lustre-MDT0001 [ 8658.693141] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 8658.757912] LustreError: 209818:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8658.764187] LustreError: 209818:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 8658.848312] Lustre: server umount lustre-MDT0001 complete [ 8660.447855] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8660.448272] LustreError: 74299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8660.455334] Lustre: Skipped 18 previous similar messages [ 8660.462231] LustreError: 74299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 118 previous similar messages [ 8677.164564] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 8677.167437] LDISKFS-fs (dm-1): recovery complete [ 8677.176883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8677.371328] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 8677.385607] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 8677.388659] Lustre: Skipped 4 previous similar messages [ 8679.128524] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8679.382389] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8679.387771] Lustre: Skipped 4 previous similar messages [ 8682.468478] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8682.472580] Lustre: Skipped 17 previous similar messages [ 8683.514018] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8683.517989] Lustre: Skipped 4 previous similar messages [ 8683.542898] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1688 to 0x2c0000400:1729) [ 8683.546792] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1687 to 0x280000400:1729) [ 8685.558537] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8686.425827] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8811.828100] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8822.979602] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 8824.011224] Lustre: Failing over lustre-MDT0000 [ 8824.026820] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8824.030701] LustreError: Skipped 3 previous similar messages [ 8824.033341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8830.044586] Lustre: server umount lustre-MDT0000 complete [ 8847.327119] Lustre: 6800:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767323792/real 1767323792] req@ffff92e303462300 x1853164275731968/t0(0) o400->MGC192.168.203.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767323808 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8847.338846] Lustre: 6800:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8847.341705] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8847.344972] LustreError: Skipped 1 previous similar message [ 8848.217501] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 8848.218848] LDISKFS-fs (dm-0): recovery complete [ 8848.222779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8856.546647] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc99e79c12 [ 8856.693756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8856.696751] Lustre: Skipped 4 previous similar messages [ 8858.422417] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8862.199854] Lustre: 212626:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 8862.204059] Lustre: 212626:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 52 previous similar messages [ 8862.703878] Lustre: 212626:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 8862.707922] Lustre: 212626:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 172 previous similar messages [ 8863.242801] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6188 to 0x2c0000401:6209) [ 8863.244374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6219 to 0x280000401:6241) [ 8865.012412] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8865.748744] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8870.231366] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 22:17:10 (1767323830) [ 8993.859863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9004.838772] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 9005.743254] Lustre: Failing over lustre-MDT0000 [ 9005.788219] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 9005.793111] Lustre: Skipped 5 previous similar messages [ 9006.044181] Lustre: server umount lustre-MDT0000 complete [ 9024.035677] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 9024.038661] LDISKFS-fs (dm-0): recovery complete [ 9024.045558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9026.148454] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9034.847672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7137 to 0x280000401:7169) [ 9034.847869] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7105 to 0x2c0000401:7137) [ 9036.792179] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9037.739067] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9163.581627] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9174.648885] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 9175.636608] Lustre: Failing over lustre-MDT0000 [ 9175.704483] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.23@tcp (stopping) [ 9175.790273] Lustre: server umount lustre-MDT0000 complete [ 9193.289943] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9193.291700] LDISKFS-fs (dm-0): recovery complete [ 9193.295914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.655762] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e43fa25500 x1853164279161984/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9204.515316] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9212.368394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8160 to 0x280000401:8193) [ 9212.369160] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8128 to 0x2c0000401:8161) [ 9214.313741] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9215.357770] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9220.177368] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 22:23:00 (1767324180) [ 9227.445039] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9229.111066] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 9229.877501] Lustre: Failing over lustre-OST0000 [ 9229.920448] Lustre: server umount lustre-OST0000 complete [ 9246.528403] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9246.530362] LDISKFS-fs (dm-2): recovery complete [ 9246.535970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9249.804986] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9253.841111] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9254.745246] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9264.807785] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 9266.449737] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 9267.284626] Lustre: Failing over lustre-OST0000 [ 9267.304493] LustreError: 219984:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9267.306867] LustreError: 219984:0:(obd_class.h:479:obd_check_dev()) Skipped 35 previous similar messages [ 9267.351736] Lustre: server umount lustre-OST0000 complete [ 9268.705820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9268.706181] LustreError: 205295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9268.710534] Lustre: Skipped 15 previous similar messages [ 9268.717030] LustreError: 205295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 108 previous similar messages [ 9283.600628] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 9283.602302] LDISKFS-fs (dm-2): recovery complete [ 9283.606038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9283.690213] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 9283.694038] Lustre: Skipped 4 previous similar messages [ 9285.389214] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9285.392623] Lustre: Skipped 4 previous similar messages [ 9285.555902] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9285.555960] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9285.562741] Lustre: Skipped 4 previous similar messages [ 9285.565153] Lustre: Skipped 17 previous similar messages [ 9286.124876] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9290.214516] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9291.209962] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9297.416805] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 22:24:18 (1767324258) [ 9421.597698] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9424.935097] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9436.085178] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 9437.157238] Lustre: Failing over lustre-MDT0001 [ 9437.254374] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 9437.463721] Lustre: server umount lustre-MDT0001 complete [ 9438.175935] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9438.178760] LustreError: Skipped 5 previous similar messages [ 9439.448569] Lustre: Failing over lustre-MDT0000 [ 9439.641539] Lustre: server umount lustre-MDT0000 complete [ 9458.415478] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9458.417455] LDISKFS-fs (dm-1): recovery complete [ 9458.426063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9459.039069] Lustre: 6799:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767324404/real 1767324404] req@ffff92e30a787800 x1853164284614400/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767324420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9459.050524] Lustre: 6799:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9459.054730] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9459.060798] LustreError: Skipped 2 previous similar messages [ 9459.064880] LustreError: 224103:0:(llog.c:1643:llog_backup()) MGC192.168.203.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 9459.068555] Lustre: 224103:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.203.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 9459.747265] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 9459.749179] LDISKFS-fs (dm-0): recovery complete [ 9459.754602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9469.215377] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e30a786680 x1853164284617856/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9469.334311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9469.339939] Lustre: Skipped 4 previous similar messages [ 9471.240560] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9471.443804] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9474.901911] LustreError: 17347:0:(ldlm_lockd.c:914:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-lustre-MDT0001_UUID lock: ffff92e30bae7e00/0xec03a6dc99ff7e0d lrc: 3/0,0 mode: PW/PW res: [0x24000272a:0x875:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x40000000000020 nid: 0@lo remote: 0xec03a6dc99ff7e06 expref: 43 pid: 17347 timeout: 0 lvb_type: 0 [ 9475.926735] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1904 to 0x280000400:1921) [ 9475.927192] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1905 to 0x2c0000400:1921) [ 9479.969760] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8198 to 0x2c0000401:8225) [ 9479.970163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8230 to 0x280000401:8257) [ 9481.680832] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9482.414507] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9483.080559] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9487.083964] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 22:27:27 (1767324447) [ 9490.179606] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9491.293673] Lustre: Failing over lustre-MDT0000 [ 9491.502866] Lustre: server umount lustre-MDT0000 complete [ 9509.008923] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 9509.010680] LDISKFS-fs (dm-0): recovery complete [ 9509.016458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9510.721407] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 9510.724448] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9526.093954] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 9526.108770] Lustre: 224129:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e30cb2ca80 x1853164220276480/t339302416754(339302416754) o101->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:113/0 lens 520/3488 e 0 to 0 dl 1767324498 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 9526.153497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8259 to 0x280000401:8289) [ 9526.153532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8198 to 0x2c0000401:8257) [ 9527.900221] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9528.609125] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9532.820301] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 22:28:13 (1767324493) [ 9536.347762] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9537.553602] Lustre: Failing over lustre-MDT0000 [ 9537.723512] Lustre: server umount lustre-MDT0000 complete [ 9554.691494] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9554.693238] LDISKFS-fs (dm-0): recovery complete [ 9554.698127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.474251] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9556.817737] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 9556.819904] LustreError: 227387:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e433a75880 x1853164220276480/t339302416754(339302416754) o101->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:144/0 lens 520/664 e 0 to 0 dl 1767324529 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 9573.192059] Lustre: lustre-MDT0000: Client c549bbb0-37cc-46d8-8645-ca0ebda68de8 (at 192.168.203.23@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 9573.199084] Lustre: 224128:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e41737d180 x1853164220276480/t339302416754(339302416754) o101->c549bbb0-37cc-46d8-8645-ca0ebda68de8@192.168.203.23@tcp:160/0 lens 520/3488 e 0 to 0 dl 1767324545 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 9573.241688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8259 to 0x280000401:8321) [ 9573.241704] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8289) [ 9574.954542] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9575.714416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9580.435203] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 22:29:00 (1767324540) [ 9582.008577] Lustre: Failing over lustre-OST0000 [ 9582.059194] Lustre: server umount lustre-OST0000 complete [ 9583.615177] Lustre: Failing over lustre-MDT0000 [ 9583.706823] Lustre: server umount lustre-MDT0000 complete [ 9599.789313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9601.483610] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9602.345949] Lustre: lustre-MDT0000: Denying connection for new client 996954a2-8a1c-40d0-9aac-3118983c1f90 (at 192.168.203.23@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 9605.107811] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8321) [ 9610.008148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9611.577384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8259 to 0x280000401:8353) [ 9612.499919] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9617.747720] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 22:29:38 (1767324578) [ 9618.297788] LustreError: 227227:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e436d5a300 x1853164285427968/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:205/0 lens 1312/4320 e 0 to 0 dl 1767324590 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 9621.597618] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9622.400909] Lustre: Failing over lustre-MDT0000 [ 9622.673699] Lustre: server umount lustre-MDT0000 complete [ 9640.992590] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9640.994559] LDISKFS-fs (dm-0): recovery complete [ 9640.999555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9642.969638] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9646.590775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8385) [ 9646.591373] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8353) [ 9647.993103] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9648.679947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9653.548699] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 22:30:13 (1767324613) [ 9654.093827] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 9654.096338] Lustre: Skipped 1 previous similar message [ 9654.098418] LustreError: 224137:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e42b7ece00 x1853164285458304/t360777252876(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:241/0 lens 2520/4320 e 0 to 0 dl 1767324626 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 9658.304040] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9661.812643] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9662.715241] Lustre: Failing over lustre-MDT0001 [ 9662.719580] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [ 9669.188169] Lustre: server umount lustre-MDT0001 complete [ 9685.094470] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9685.096282] LDISKFS-fs (dm-1): recovery complete [ 9685.103746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9686.762288] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9690.612645] Lustre: 233193:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e41754b480 x1853164224140032/t25769805143(0) o36->996954a2-8a1c-40d0-9aac-3118983c1f90@192.168.203.23@tcp:308/0 lens 560/2880 e 0 to 0 dl 1767324693 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9690.615904] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1932 to 0x2c0000400:1953) [ 9690.615904] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1932 to 0x280000400:1953) [ 9692.021016] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9692.652456] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9696.882215] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 22:30:57 (1767324657) [ 9700.765775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9704.023831] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9704.832392] Lustre: Failing over lustre-MDT0000 [ 9705.141404] Lustre: server umount lustre-MDT0000 complete [ 9722.663319] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9722.664802] LDISKFS-fs (dm-0): recovery complete [ 9722.667889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.552054] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e434108700 x1853164285507200/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9733.374915] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9736.769804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8385) [ 9736.770150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8417) [ 9738.541931] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9739.454122] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9741.651947] Lustre: Failing over lustre-MDT0001 [ 9741.768387] Lustre: server umount lustre-MDT0001 complete [ 9757.792063] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9757.793751] LDISKFS-fs (dm-1): recovery complete [ 9757.798580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9759.414587] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9763.316982] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1964 to 0x2c0000400:1985) [ 9763.317018] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1964 to 0x280000400:1985) [ 9764.705229] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9765.407616] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9769.680111] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 22:32:10 (1767324730) [ 9770.120887] LustreError: 224136:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e409ee5500 x1853164285546368/t365072220176(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:357/0 lens 2520/4320 e 0 to 0 dl 1767324742 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 9770.131035] LustreError: 224136:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 9776.204674] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9779.436321] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9780.230674] Lustre: Failing over lustre-MDT0000 [ 9780.478354] Lustre: server umount lustre-MDT0000 complete [ 9781.887109] LustreError: 15893:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324743 with bad export cookie 17006620084287843821 [ 9781.888764] Lustre: Failing over lustre-MDT0001 [ 9781.892196] LustreError: 15893:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9787.872241] LustreError: 241656:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff92e43315b800 x1853164285557120/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 9787.879366] LustreError: 241656:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 9787.985789] Lustre: server umount lustre-MDT0001 complete [ 9805.024449] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9805.028881] LDISKFS-fs (dm-0): recovery complete [ 9805.038945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9806.043381] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9806.045153] LDISKFS-fs (dm-1): recovery complete [ 9806.050580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9807.135115] LustreError: 243042:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 9807.138635] LustreError: 243042:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff92e409ee6680 x1853164285557248/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1767324768 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9807.146051] LustreError: 243042:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 9807.327332] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e409ee5500 x1853164285557888/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9809.085373] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9809.206631] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9812.732269] Lustre: 243945:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e301ad1f80 x1853164224211840/t34359738438(0) o36->996954a2-8a1c-40d0-9aac-3118983c1f90@192.168.203.23@tcp:426/0 lens 560/2880 e 0 to 0 dl 1767324811 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9812.741879] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1996 to 0x280000400:2017) [ 9812.741880] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1996 to 0x2c0000400:2017) [ 9825.126808] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8417) [ 9825.126863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8449) [ 9827.076780] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9827.936629] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9828.679453] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9832.966113] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 22:33:13 (1767324793) [ 9833.437195] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 9833.439478] Lustre: Skipped 2 previous similar messages [ 9839.647590] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9840.372300] Lustre: Failing over lustre-MDT0000 [ 9840.764809] Lustre: server umount lustre-MDT0000 complete [ 9857.699085] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9857.701152] LDISKFS-fs (dm-0): recovery complete [ 9857.710103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.465034] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9863.161213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8481) [ 9863.161269] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8449) [ 9864.596731] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9865.228330] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9888.580639] Lustre: lustre-MDT0001: Client 996954a2-8a1c-40d0-9aac-3118983c1f90 (at 192.168.203.23@tcp) reconnecting [ 9888.583732] Lustre: Skipped 2 previous similar messages [ 9888.586941] Lustre: 243945:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e30694d500 x1853164224245376/t38654705734(0) o36->996954a2-8a1c-40d0-9aac-3118983c1f90@192.168.203.23@tcp:515/0 lens 560/2880 e 0 to 0 dl 1767324900 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9891.399912] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 22:34:12 (1767324852) [ 9894.519060] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9895.237768] Lustre: Failing over lustre-MDT0001 [ 9895.299629] LustreError: 247064:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9895.302472] LustreError: 247064:0:(obd_class.h:479:obd_check_dev()) Skipped 103 previous similar messages [ 9895.327680] Lustre: server umount lustre-MDT0001 complete [ 9896.769355] LustreError: 243945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.776320] LustreError: 243945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 178 previous similar messages [ 9898.975953] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9898.980875] Lustre: Skipped 42 previous similar messages [ 9910.806895] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9910.808667] LDISKFS-fs (dm-1): recovery complete [ 9910.815174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9910.942347] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 9910.945430] Lustre: Skipped 13 previous similar messages [ 9912.127629] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9912.130875] Lustre: Skipped 13 previous similar messages [ 9912.331957] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9916.386301] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 9916.390245] Lustre: Skipped 44 previous similar messages [ 9916.395866] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9916.403211] Lustre: Skipped 13 previous similar messages [ 9916.421440] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2038 to 0x280000400:2081) [ 9916.421622] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2038 to 0x2c0000400:2081) [ 9917.718259] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9918.356806] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9922.419936] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 22:34:43 (1767324883) [ 9922.833165] LustreError: 243204:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e436d97480 x1853164224310272/t42949673030(0) o36->996954a2-8a1c-40d0-9aac-3118983c1f90@192.168.203.23@tcp:510/0 lens 560/448 e 0 to 0 dl 1767324895 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 9922.843032] LustreError: 243204:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 2 previous similar messages [ 9928.581479] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9931.322170] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9932.133132] Lustre: Failing over lustre-MDT0000 [ 9932.299892] Lustre: server umount lustre-MDT0000 complete [ 9948.579165] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 9948.581011] LDISKFS-fs (dm-0): recovery complete [ 9948.585289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9950.288852] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9953.802635] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8481) [ 9953.802682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8513) [ 9955.242749] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9955.878649] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9957.961587] Lustre: Failing over lustre-MDT0001 [ 9958.041656] Lustre: server umount lustre-MDT0001 complete [ 9973.376939] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 9973.378707] LDISKFS-fs (dm-1): recovery complete [ 9973.383342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9974.908101] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9978.870328] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2092 to 0x280000400:2113) [ 9978.870342] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2092 to 0x2c0000400:2113) [ 9980.025133] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9980.600748] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9984.280992] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 22:35:45 (1767324945) [ 9990.104520] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9992.700970] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9993.340809] Lustre: Failing over lustre-MDT0000 [ 9993.582653] Lustre: server umount lustre-MDT0000 complete [ 9994.768738] LustreError: 14419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324956 with bad export cookie 17006620084287857576 [ 9994.769408] Lustre: Failing over lustre-MDT0001 [ 9994.772372] LustreError: 14419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9994.873739] Lustre: server umount lustre-MDT0001 complete [10011.143320] LDISKFS-fs (dm-0): 2 truncates cleaned up [10011.144438] LDISKFS-fs (dm-0): recovery complete [10011.146744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10012.074242] LDISKFS-fs (dm-1): 6 truncates cleaned up [10012.075564] LDISKFS-fs (dm-1): recovery complete [10012.079541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10019.807444] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e433d74380 x1853164285729152/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10021.295628] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10021.364098] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10025.134765] Lustre: 255119:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e4333d3b80 x1853164224347904/t47244640326(0) o36->996954a2-8a1c-40d0-9aac-3118983c1f90@192.168.203.23@tcp:612/0 lens 560/2880 e 0 to 0 dl 1767324997 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [10025.139788] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2124 to 0x2c0000400:2145) [10025.139813] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2124 to 0x280000400:2145) [10025.143643] Lustre: 255119:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 2 previous similar messages [10034.010299] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8513) [10034.010302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8545) [10035.400674] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10035.974684] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10036.485331] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10039.994211] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 22:36:40 (1767325000) [10042.708176] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10043.350937] Lustre: Failing over lustre-MDT0001 [10043.356371] LustreError: 256801:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x240002ef9:0xa9:0x0].0x0 (ffff92e40ccb4f00) refcount nonzero (1) after lock cleanup; forcing cleanup. [10043.367682] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.23@tcp (stopping) [10043.370016] Lustre: Skipped 12 previous similar messages [10049.069265] Lustre: server umount lustre-MDT0001 complete [10064.084524] LDISKFS-fs (dm-1): 6 truncates cleaned up [10064.086425] LDISKFS-fs (dm-1): recovery complete [10064.092751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10065.547917] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10069.491511] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2156 to 0x280000400:2177) [10069.491581] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2156 to 0x2c0000400:2177) [10070.703665] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10071.284789] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10074.678817] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 22:37:15 (1767325035) [10077.474786] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10078.082883] Lustre: Failing over lustre-MDT0000 [10078.396109] Lustre: server umount lustre-MDT0000 complete [10079.712379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10079.715232] LustreError: Skipped 15 previous similar messages [10090.463187] Lustre: 257571:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767325036/real 1767325036] req@ffff92e30c441180 x1853164285789952/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1488/4320 e 0 to 1 dl 1767325052 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [10090.471690] Lustre: 257571:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [10095.967083] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10095.969297] LustreError: Skipped 9 previous similar messages [10096.140907] LDISKFS-fs (dm-0): 2 truncates cleaned up [10096.142117] LDISKFS-fs (dm-0): recovery complete [10096.144696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10106.396508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10106.400108] Lustre: Skipped 18 previous similar messages [10107.703641] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10111.462699] Lustre: 259741:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [10111.467152] Lustre: 259741:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 202 previous similar messages [10111.489777] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8545) [10111.489815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8577) [10112.799778] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10113.351661] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10117.374342] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 22:37:58 (1767325078) [10117.846439] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [10117.848915] Lustre: Skipped 5 previous similar messages [10120.732295] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10123.485539] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10124.129587] Lustre: Failing over lustre-MDT0000 [10124.448314] Lustre: server umount lustre-MDT0000 complete [10140.280087] LDISKFS-fs (dm-0): 2 truncates cleaned up [10140.281327] LDISKFS-fs (dm-0): recovery complete [10140.284539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10141.695992] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10145.791637] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8577) [10145.791643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8609) [10147.015630] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10147.567714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10149.578923] Lustre: Failing over lustre-MDT0001 [10149.676433] Lustre: server umount lustre-MDT0001 complete [10164.889344] LDISKFS-fs (dm-1): 6 truncates cleaned up [10164.890636] LDISKFS-fs (dm-1): recovery complete [10164.894183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10166.410989] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10170.354523] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2156 to 0x2c0000400:2209) [10170.354649] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2156 to 0x280000400:2209) [10171.551409] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10172.102502] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10175.532085] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 22:38:56 (1767325136) [10178.501069] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10181.146569] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10181.810675] Lustre: Failing over lustre-MDT0000 [10182.039127] Lustre: server umount lustre-MDT0000 complete [10183.331616] LustreError: 14421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325144 with bad export cookie 17006620084287867971 [10183.333053] Lustre: Failing over lustre-MDT0001 [10183.334756] LustreError: 14421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10183.336948] LustreError: 265453:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200028c71:0x1f:0x0].0xf7117594 (ffff92e40b6c0900) refcount nonzero (1) after lock cleanup; forcing cleanup. [10183.341302] LustreError: 265453:0:(ldlm_resource.c:981:ldlm_resource_complain()) Skipped 2 previous similar messages [10188.768058] LustreError: 265453:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff92e30d0eb480 x1853164285871360/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [10188.774628] LustreError: 265453:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [10188.875086] Lustre: server umount lustre-MDT0001 complete [10204.878396] LDISKFS-fs (dm-0): 2 truncates cleaned up [10204.880082] LDISKFS-fs (dm-0): recovery complete [10204.882871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10205.856965] LDISKFS-fs (dm-1): 6 truncates cleaned up [10205.858603] LDISKFS-fs (dm-1): recovery complete [10205.862922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10208.223418] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e4330e3800 x1853164285872128/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10208.230441] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [10209.826169] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10209.966889] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10212.593798] Lustre: 267743:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff92e4347a2d80 x1853164224427392/t60129542146(0) o36->996954a2-8a1c-40d0-9aac-3118983c1f90@192.168.203.23@tcp:70/0 lens 496/2888 e 0 to 0 dl 1767325210 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [10212.598794] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2156 to 0x280000400:2241) [10212.599051] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2156 to 0x2c0000400:2241) [10212.603082] Lustre: 267743:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [10226.523528] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8609) [10226.523527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8641) [10227.853236] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10228.435984] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10228.952121] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10232.318053] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 22:39:53 (1767325193) [10232.673867] LustreError: 267743:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff92e436c57800 x1853164224442112/t64424509442(0) o36->996954a2-8a1c-40d0-9aac-3118983c1f90@192.168.203.23@tcp:65/0 lens 496/456 e 0 to 0 dl 1767325205 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [10232.684481] LustreError: 267743:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 5 previous similar messages [10235.308863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10235.932924] Lustre: Failing over lustre-MDT0000 [10236.141729] Lustre: server umount lustre-MDT0000 complete [10247.999752] Lustre: lustre-MDT0001: Client 996954a2-8a1c-40d0-9aac-3118983c1f90 (at 192.168.203.23@tcp) reconnecting [10248.003180] Lustre: Skipped 1 previous similar message [10251.819301] LDISKFS-fs (dm-0): 2 truncates cleaned up [10251.820714] LDISKFS-fs (dm-0): recovery complete [10251.823584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10253.124722] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10257.401950] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8641) [10257.402194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8673) [10258.542628] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10259.089470] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10262.491880] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 22:40:23 (1767325223) [10265.325112] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10265.934115] Lustre: Failing over lustre-MDT0001 [10266.020362] Lustre: server umount lustre-MDT0001 complete [10281.228725] LDISKFS-fs (dm-1): 6 truncates cleaned up [10281.230440] LDISKFS-fs (dm-1): recovery complete [10281.236715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10282.685509] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10286.584562] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2156 to 0x2c0000400:2273) [10286.584574] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2156 to 0x280000400:2273) [10287.862843] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10288.388097] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10291.805050] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 22:40:52 (1767325252) [10294.610385] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10297.000826] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10297.628232] Lustre: Failing over lustre-MDT0000 [10297.835530] Lustre: server umount lustre-MDT0000 complete [10313.513029] LDISKFS-fs (dm-0): 2 truncates cleaned up [10313.514737] LDISKFS-fs (dm-0): recovery complete [10313.518126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10314.909968] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10318.844189] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8673) [10318.844213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8705) [10319.937336] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10320.453240] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10322.404865] Lustre: Failing over lustre-MDT0001 [10322.480529] Lustre: server umount lustre-MDT0001 complete [10337.268759] LDISKFS-fs (dm-1): 6 truncates cleaned up [10337.270739] LDISKFS-fs (dm-1): recovery complete [10337.274165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10338.611624] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10342.391961] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2156 to 0x2c0000400:2305) [10342.391966] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2156 to 0x280000400:2305) [10343.546659] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10344.077114] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10347.332101] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 22:41:48 (1767325308) [10350.027472] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10352.422164] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10353.009779] Lustre: Failing over lustre-MDT0000 [10353.232902] Lustre: server umount lustre-MDT0000 complete [10354.356591] LustreError: 14421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325315 with bad export cookie 17006620084287875503 [10354.357976] Lustre: Failing over lustre-MDT0001 [10354.360090] LustreError: 14421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10354.463102] Lustre: server umount lustre-MDT0001 complete [10370.416204] LDISKFS-fs (dm-0): 2 truncates cleaned up [10370.418063] LDISKFS-fs (dm-0): recovery complete [10370.421089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.427381] LDISKFS-fs (dm-1): 6 truncates cleaned up [10371.429527] LDISKFS-fs (dm-1): recovery complete [10371.434993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10379.232513] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xec03a6dc9a00b333 [10380.705937] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10380.778806] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10388.308300] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:2156 to 0x280000400:2337) [10388.308300] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:2156 to 0x2c0000400:2337) [10393.441334] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8259 to 0x2c0000401:8705) [10393.441343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8355 to 0x280000401:8737) [10394.933404] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10395.462127] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10395.968671] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10399.347910] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 22:42:40 (1767325360) [10399.903669] Lustre: 280174:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 996954a2-8a1c-40d0-9aac-3118983c1f90 at adminstrative request [10405.706956] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 22:42:46 (1767325366) [10407.622800] Lustre: Failing over lustre-MDT0000 [10407.875151] Lustre: server umount lustre-MDT0000 complete [10423.055963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10424.592411] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10428.448947] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8756 to 0x2c0000401:8801) [10428.448982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8789 to 0x280000401:8833) [10429.700249] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10430.362868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10434.034560] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 22:43:14 (1767325394) [10438.564080] Lustre: Failing over lustre-OST0000 [10438.629802] Lustre: server umount lustre-OST0000 complete [10452.379290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10454.311193] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10457.474657] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10458.020595] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10461.736678] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 22:43:42 (1767325422) [10463.100450] Lustre: Failing over lustre-MDT0000 [10463.337775] Lustre: server umount lustre-MDT0000 complete [10465.882547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10467.303783] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10468.008237] Lustre: lustre-MDT0000: Denying connection for new client 1901dc20-7284-4c6f-a51e-388bba0ad678 (at 192.168.203.23@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [10468.014359] Lustre: Skipped 1 previous similar message [10471.416929] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8789 to 0x280000401:8865) [10471.416932] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8756 to 0x2c0000401:8833) [10475.640472] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 22:43:56 (1767325436) [10478.462949] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [10479.365935] Lustre: Failing over lustre-OST0000 [10479.399616] Lustre: server umount lustre-OST0000 complete [10494.491148] LDISKFS-fs (dm-2): 3 truncates cleaned up [10494.492806] LDISKFS-fs (dm-2): recovery complete [10494.496222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10496.484600] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10499.525250] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10500.060714] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10503.540449] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 22:44:24 (1767325464) [10506.330801] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [10508.235947] Lustre: Failing over lustre-OST0000 [10508.250137] LustreError: 287073:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [10508.252104] LustreError: 287073:0:(obd_class.h:479:obd_check_dev()) Skipped 159 previous similar messages [10508.274714] Lustre: server umount lustre-OST0000 complete [10509.125136] LustreError: 205293:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10509.130730] LustreError: 205293:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 216 previous similar messages [10509.792118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10509.795868] Lustre: Skipped 61 previous similar messages [10523.411992] LDISKFS-fs (dm-2): 3 truncates cleaned up [10523.414308] LDISKFS-fs (dm-2): recovery complete [10523.418421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10523.476780] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [10523.480035] Lustre: Skipped 20 previous similar messages [10524.479748] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [10524.483241] Lustre: Skipped 20 previous similar messages [10524.604522] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.23@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:8867 extent [0-4194303]: client csum 34fd51a2, server csum 89f56ea3 [10524.650832] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [10524.650871] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [10524.653337] Lustre: Skipped 68 previous similar messages [10524.658908] Lustre: Skipped 20 previous similar messages [10525.428553] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10528.548386] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10529.075542] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10532.623103] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 22:44:53 (1767325493) [10535.183590] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [10537.778124] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10539.803095] Lustre: Failing over lustre-MDT0000 [10540.059400] Lustre: server umount lustre-MDT0000 complete [10541.412828] LustreError: 57348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325503 with bad export cookie 17006620084287901592 [10541.416691] Lustre: Failing over lustre-OST0000 [10541.417900] LustreError: 57348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10543.484267] Lustre: server umount lustre-OST0000 complete [10558.581721] LDISKFS-fs (dm-0): 2 truncates cleaned up [10558.582947] LDISKFS-fs (dm-0): recovery complete [10558.585605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10566.623435] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff92e43fa38e00 x1853164286154368/t0(0) o250->MGC192.168.203.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10566.631968] LustreError: 6796:0:(client.c:1390:ptlrpc_import_delay_req()) Skipped 4 previous similar messages [10567.955291] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10571.870250] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8756 to 0x2c0000401:8865) [10582.302672] LDISKFS-fs (dm-2): 3 truncates cleaned up [10582.304654] LDISKFS-fs (dm-2): recovery complete [10582.308119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10584.200425] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10589.788582] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 22:45:50 (1767325550) [10594.556591] Lustre: Failing over lustre-OST0000 [10594.616403] Lustre: server umount lustre-OST0000 complete [10595.797927] Lustre: Failing over lustre-MDT0000 [10596.043540] Lustre: server umount lustre-MDT0000 complete [10610.534294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10612.025810] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10615.800810] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8756 to 0x2c0000401:8897) [10619.751721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10621.759220] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10622.534463] Lustre: lustre-OST0000: Denying connection for new client 8085a37c-f716-4e3c-be61-d108e0477003 (at 192.168.203.23@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [10627.902600] Lustre: lustre-OST0000: Denying connection for new client 8085a37c-f716-4e3c-be61-d108e0477003 (at 192.168.203.23@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [10627.907188] Lustre: Skipped 1 previous similar message [10638.142782] Lustre: lustre-OST0000: Denying connection for new client 8085a37c-f716-4e3c-be61-d108e0477003 (at 192.168.203.23@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [10638.147175] Lustre: Skipped 1 previous similar message [10658.622820] Lustre: lustre-OST0000: Denying connection for new client 8085a37c-f716-4e3c-be61-d108e0477003 (at 192.168.203.23@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:31 [10658.627522] Lustre: Skipped 3 previous similar messages [10690.500141] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [10690.502476] Lustre: 294340:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 1901dc20-7284-4c6f-a51e-388bba0ad678@ [10690.506689] Lustre: 294340:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [10690.509359] Lustre: lustre-OST0000: disconnecting 1 stale clients [10690.516599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8916 to 0x280000401:8937) [10696.015345] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [10700.531522] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [10702.677842] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 22:47:43 (1767325663) [10704.112439] Lustre: Failing over lustre-OST0000 [10704.153146] Lustre: server umount lustre-OST0000 complete [10705.887672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10705.891256] LustreError: Skipped 16 previous similar messages [10717.277479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10717.335929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10717.338381] Lustre: Skipped 19 previous similar messages [10719.184582] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10723.195976] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 22:48:04 (1767325684) [10724.455094] Lustre: Failing over lustre-OST0000 [10724.497696] Lustre: server umount lustre-OST0000 complete [10737.937967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10739.373058] LustreError: 297461:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [10739.376796] LustreError: 297461:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 77 previous similar messages [10739.788037] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10745.311312] Lustre: *** cfs_fail_loc=715, val=40*** [10754.527130] Lustre: 6796:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767325700/real 1767325700] req@ffff92e40568c700 x1853164286262272/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767325716 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [10754.536860] Lustre: 6796:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [10754.540449] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [10754.543318] Lustre: Skipped 1 previous similar message [10760.671185] Lustre: *** cfs_fail_loc=715, val=40*** [10760.673603] Lustre: Skipped 2 previous similar messages [10770.911446] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [10770.914039] Lustre: Skipped 2 previous similar messages [10777.055140] Lustre: *** cfs_fail_loc=715, val=40*** [10777.056295] Lustre: Skipped 2 previous similar messages [10779.423145] LustreError: 297461:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [10779.426522] LustreError: 297461:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 77 previous similar messages [10780.725489] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10781.271574] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10784.549462] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 22:49:05 (1767325745) [10785.740791] Lustre: Failing over lustre-MDT0000 [10786.007156] Lustre: server umount lustre-MDT0000 complete [10800.457420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.490578] LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10800.493604] LustreError: Skipped 9 previous similar messages [10801.782514] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10805.733808] LustreError: 299184:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [10808.287188] Lustre: *** cfs_fail_loc=715, val=80*** [10808.288384] Lustre: Skipped 2 previous similar messages [10818.366819] Lustre: lustre-MDT0000: Client 8085a37c-f716-4e3c-be61-d108e0477003 (at 192.168.203.23@tcp) reconnected, waiting for 2 clients in recovery for 1:28 [10818.371245] Lustre: Skipped 1 previous similar message [10822.111457] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10824.671113] Lustre: *** cfs_fail_loc=715, val=80*** [10824.672581] Lustre: Skipped 1 previous similar message [10834.750561] Lustre: lustre-MDT0000: Client 8085a37c-f716-4e3c-be61-d108e0477003 (at 192.168.203.23@tcp) reconnected, waiting for 2 clients in recovery for 1:11 [10837.471936] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10841.055124] Lustre: *** cfs_fail_loc=715, val=80*** [10841.056696] Lustre: Skipped 1 previous similar message [10851.134632] Lustre: lustre-MDT0000: Client 8085a37c-f716-4e3c-be61-d108e0477003 (at 192.168.203.23@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [10853.855429] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10857.439112] Lustre: *** cfs_fail_loc=715, val=80*** [10857.440449] Lustre: Skipped 1 previous similar message [10867.518768] Lustre: lustre-MDT0000: Client 8085a37c-f716-4e3c-be61-d108e0477003 (at 192.168.203.23@tcp) reconnected, waiting for 2 clients in recovery for 0:38 [10870.239781] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10885.599515] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [10885.815074] LustreError: 299184:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [10885.819771] Lustre: 299184:0:(ldlm_lib.c:2931:target_recovery_thread()) too long recovery - read logs [10885.822502] LustreError: dumping log to /tmp/lustre-log.1767325847.299184 [10885.881426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8951 to 0x280000401:8969) [10885.881491] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8910 to 0x2c0000401:8929) [10887.146295] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10887.674863] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10891.220037] Lustre: DEBUG MARKER: == replay-single test complete, duration 10359 sec ======= 22:50:52 (1767325852) [10891.720718] Lustre: DEBUG MARKER: === replay-single: start cleanup 22:50:52 (1767325852) === [10894.088910] Lustre: DEBUG MARKER: === replay-single: finish cleanup 22:50:54 (1767325854) === [10894.727039] Lustre: Failing over lustre-MDT0000 [10894.880592] Lustre: server umount lustre-MDT0000 complete [10911.410243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.734900] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10916.853808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8951 to 0x280000401:9001) [10916.853837] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8910 to 0x2c0000401:8961) [10917.990585] Lustre: DEBUG MARKER: oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10918.531392] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10927.072061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10927.073708] Lustre: Skipped 14 previous similar messages [10930.387178] Lustre: server umount lustre-MDT0000 complete [10933.154847] LustreError: 14419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325894 with bad export cookie 17006620084287934702 [10933.158149] LustreError: 14419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10933.262407] Lustre: server umount lustre-MDT0001 complete [10946.559366] Lustre: server umount lustre-OST0000 complete [10959.462336] Lustre: server umount lustre-OST0001 complete [10965.602763] Lustre: DEBUG MARKER: oleg323-server.virtnet: executing unload_modules_local [10966.610361] Key type lgssc unregistered [10966.740250] LNet: 304331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10966.742304] LNetError: 304331:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10966.749276] LNet: Removed LNI 192.168.203.123@tcp [10967.062094] Key type .llcrypt unregistered [10967.063277] Key type ._llcrypt unregistered