[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-22) (GCC)) #7 SMP Sat Jan 18 21:01:29 EST 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1714863910 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcbe3000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 513 pages/cpu s2064384 r8192 d28672 u4194304 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2894788K/4306352K available (20483K kernel code, 12066K rwdata, 7356K rodata, 4680K init, 23504K bss, 542476K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41388 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x616/0x99a with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar [ 0.001000] ... MAX_LOCKDEP_SUBCLASSES: 8 [ 0.001000] ... MAX_LOCK_DEPTH: 48 [ 0.001000] ... MAX_LOCKDEP_KEYS: 8192 [ 0.001000] ... CLASSHASH_SIZE: 4096 [ 0.001000] ... MAX_LOCKDEP_ENTRIES: 32768 [ 0.001000] ... MAX_LOCKDEP_CHAINS: 65536 [ 0.001000] ... CHAINHASH_SIZE: 32768 [ 0.001000] memory used by lock dependency info: 4149 kB [ 0.001000] per task-struct memory footprint: 2688 bytes [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002015] Switched APIC routing to physical x2apic. [ 0.003018] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008020] pid_max: default: 32768 minimum: 301 [ 0.009405] LSM: Security Framework initializing [ 0.010194] Yama: becoming mindful. [ 0.011144] SELinux: Initializing. [ 0.013125] *** VALIDATE selinux *** [ 0.022367] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028345] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029218] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031076] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032307] *** VALIDATE tmpfs *** [ 0.035104] *** VALIDATE proc *** [ 0.037658] *** VALIDATE cgroup *** [ 0.038020] *** VALIDATE cgroup2 *** [ 0.039000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039242] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041046] Spectre V2 : User space: Vulnerable [ 0.042017] Speculative Store Bypass: Vulnerable [ 0.048085] debug: unmapping init [mem 0xffffffff96503000-0xffffffff9650afff] [ 0.051222] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.054103] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.055050] ... version: 2 [ 0.056013] ... bit width: 48 [ 0.057024] ... generic registers: 4 [ 0.058014] ... value mask: 0000ffffffffffff [ 0.059028] ... max period: 00007fffffffffff [ 0.060029] ... fixed-purpose events: 3 [ 0.061019] ... event mask: 000000070000000f [ 0.062637] rcu: Hierarchical SRCU implementation. [ 0.068466] smp: Bringing up secondary CPUs ... [ 0.071391] x86: Booting SMP configuration: [ 0.072025] .... node #0, CPUs: #1 [ 0.080814] #2 [ 0.090813] #3 [ 0.096022] smp: Brought up 1 node, 4 CPUs [ 0.097038] smpboot: Max logical packages: 1 [ 0.098020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143833] node 0 deferred pages initialised in 36ms [ 0.147489] pgdatinit0 (35) used greatest stack depth: 14528 bytes left [ 0.159028] devtmpfs: initialized [ 0.162234] x86/mm: Memory block size: 128MB [ 0.185858] gcov: version magic: 0x41383552 [ 0.191244] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.192162] futex hash table entries: 1024 (order: 5, 131072 bytes, vmalloc) [ 0.193757] pinctrl core: initialized pinctrl subsystem [ 0.195408] [ 0.196022] ************************************************************* [ 0.197018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198023] ** ** [ 0.199023] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.200025] ** ** [ 0.201012] ** This means that this kernel is built to expose internal ** [ 0.202034] ** IOMMU data structures, which may compromise security on ** [ 0.203024] ** your system. ** [ 0.204018] ** ** [ 0.205018] ** If you see this message and you are not debugging the ** [ 0.206036] ** kernel, report this immediately to your vendor! ** [ 0.207020] ** ** [ 0.208024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209025] ************************************************************* [ 0.213238] NET: Registered protocol family 16 [ 0.216933] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.217127] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.219069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.223165] cpuidle: using governor menu [ 0.226478] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.245678] PCI: Using configuration type 1 for base access [ 0.246000] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.385847] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.386023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.403412] cryptd: max_cpu_qlen set to 1000 [ 0.409027] ACPI: Added _OSI(Module Device) [ 0.411025] ACPI: Added _OSI(Processor Device) [ 0.414024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.417025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.468549] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.493154] ACPI: Interpreter enabled [ 0.494273] ACPI: PM: (supports S0 S3 S4 S5) [ 0.496020] ACPI: Using IOAPIC for interrupt routing [ 0.497292] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.505953] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.636917] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.644089] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.649028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.660354] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.700000] acpiphp: Slot [2] registered [ 0.713216] acpiphp: Slot [5] registered [ 0.714323] acpiphp: Slot [6] registered [ 0.720369] acpiphp: Slot [7] registered [ 0.724360] acpiphp: Slot [8] registered [ 0.730430] acpiphp: Slot [9] registered [ 0.737454] acpiphp: Slot [10] registered [ 0.748462] acpiphp: Slot [3] registered [ 0.751313] acpiphp: Slot [4] registered [ 0.755000] acpiphp: Slot [11] registered [ 0.756010] acpiphp: Slot [12] registered [ 0.761031] acpiphp: Slot [13] registered [ 0.765351] acpiphp: Slot [14] registered [ 0.766295] acpiphp: Slot [15] registered [ 0.773439] acpiphp: Slot [16] registered [ 0.774350] acpiphp: Slot [17] registered [ 0.783230] acpiphp: Slot [18] registered [ 0.784429] acpiphp: Slot [19] registered [ 0.805165] acpiphp: Slot [20] registered [ 0.806405] acpiphp: Slot [21] registered [ 0.825443] acpiphp: Slot [22] registered [ 0.827351] acpiphp: Slot [23] registered [ 0.838385] acpiphp: Slot [24] registered [ 0.839299] acpiphp: Slot [25] registered [ 0.841000] acpiphp: Slot [26] registered [ 0.852308] acpiphp: Slot [27] registered [ 0.853358] acpiphp: Slot [28] registered [ 0.872381] acpiphp: Slot [29] registered [ 0.874302] acpiphp: Slot [30] registered [ 0.888476] acpiphp: Slot [31] registered [ 0.891189] PCI host bridge to bus 0000:00 [ 0.898060] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.900041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.923076] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.940098] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.947054] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.949049] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.961407] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.976000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 1.003000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 1.040029] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 1.050000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.058028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.071031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.073019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.106812] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.119718] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.122057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.136113] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 15625 usecs [ 1.170258] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 1.183014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 1.227785] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 1.235017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 1.312000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.339022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.363022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.408023] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 1.471254] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.484019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.507021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.541029] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 1.608475] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.629019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.639019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.682037] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 1.737426] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.750019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.757019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.772043] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 1.801098] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.812023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.816016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.827017] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 1.871267] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.878016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.883000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.900027] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 1.953437] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.957000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.963000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.968231] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.974972] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.993000] iommu: Default domain type: Passthrough [ 1.995126] SCSI subsystem initialized [ 2.001720] ACPI: bus type USB registered [ 2.004466] usbcore: registered new interface driver usbfs [ 2.007331] usbcore: registered new interface driver hub [ 2.011291] usbcore: registered new device driver usb [ 2.016572] pps_core: LinuxPPS API ver. 1 registered [ 2.017000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 2.018132] PTP clock support registered [ 2.027791] EDAC MC: Ver: 3.0.0 [ 2.035504] PCI: Using ACPI for IRQ routing [ 2.038831] NetLabel: Initializing [ 2.041017] NetLabel: domain hash size = 128 [ 2.042010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 2.043000] NetLabel: unlabeled traffic allowed by default [ 2.044159] vgaarb: loaded [ 2.049445] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 2.051015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 2.070509] clocksource: Switched to clocksource kvm-clock [ 3.155237] VFS: Disk quotas dquot_6.6.0 [ 3.157808] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 3.162814] *** VALIDATE ramfs *** [ 3.164378] *** VALIDATE hugetlbfs *** [ 3.169078] pnp: PnP ACPI init [ 3.185721] pnp: PnP ACPI: found 6 devices [ 3.257411] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 3.261321] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 3.263604] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 3.265945] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 3.268440] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 3.270973] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 3.275096] NET: Registered protocol family 2 [ 3.278703] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 3.285566] tcp_listen_portaddr_hash hash table entries: 4096 (order: 6, 360448 bytes, vmalloc) [ 3.290519] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 3.298896] TCP bind hash table entries: 65536 (order: 10, 5242880 bytes, vmalloc) [ 3.306832] TCP: Hash tables configured (established 65536 bind 65536) [ 3.312828] MPTCP token hash table entries: 8192 (order: 7, 786432 bytes, vmalloc) [ 3.317181] UDP hash table entries: 4096 (order: 7, 786432 bytes, vmalloc) [ 3.320995] UDP-Lite hash table entries: 4096 (order: 7, 786432 bytes, vmalloc) [ 3.325518] NET: Registered protocol family 1 [ 3.334312] RPC: Registered named UNIX socket transport module. [ 3.343158] RPC: Registered udp transport module. [ 3.344702] RPC: Registered tcp transport module. [ 3.351454] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 3.354587] NET: Registered protocol family 44 [ 3.360647] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 3.362818] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 3.365346] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 3.369849] PCI: CLS 0 bytes, default 64 [ 3.376372] Unpacking initramfs... [ 9.500054] debug: unmapping init [mem 0xffff9e95bcbe3000-0xffff9e95bffbffff] [ 9.515214] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 9.534450] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 9.544907] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 9.588247] cryptomgr_test (64) used greatest stack depth: 14248 bytes left [ 10.199065] hrtimer: interrupt took 12052117 ns [ 13.590956] Initialise system trusted keyrings [ 13.595358] Key type blacklist registered [ 13.600562] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 13.807289] zbud: loaded [ 13.888854] *** VALIDATE nfs *** [ 13.890651] *** VALIDATE nfs4 *** [ 13.896234] pstore: using deflate compression [ 13.916521] Platform Keyring initialized [ 13.925814] cryptomgr_test (72) used greatest stack depth: 14024 bytes left [ 13.981938] cryptomgr_test (83) used greatest stack depth: 13976 bytes left [ 14.020073] cryptomgr_test (85) used greatest stack depth: 13800 bytes left [ 14.123414] cryptomgr_test (93) used greatest stack depth: 13640 bytes left [ 14.331476] cryptomgr_test (118) used greatest stack depth: 13592 bytes left [ 14.349536] NET: Registered protocol family 38 [ 14.351215] Key type asymmetric registered [ 14.352619] Asymmetric key parser 'x509' registered [ 14.361711] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 14.372769] io scheduler mq-deadline registered [ 14.374447] io scheduler kyber registered [ 14.451898] io scheduler bfq registered [ 14.462949] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 14.470932] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 14.474649] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 14.479273] ACPI: Power Button [PWRF] [ 21.356325] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 26.984638] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 39.646731] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 46.657431] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 59.954732] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 60.127116] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 60.273441] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 60.356042] Non-volatile memory driver v1.3 [ 60.357575] Linux agpgart interface v0.103 [ 61.044120] virtio_blk virtio1: [vda] 131896 512-byte logical blocks (67.5 MB/64.4 MiB) [ 61.067282] vda: detected capacity change from 0 to 67530752 [ 61.253053] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 61.264477] vdb: detected capacity change from 0 to 1073741824 [ 61.479431] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 61.499187] vdc: detected capacity change from 0 to 2621440000 [ 61.624230] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 61.642732] vdd: detected capacity change from 0 to 2621440000 [ 61.818117] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 61.846994] vde: detected capacity change from 0 to 4294967296 [ 62.005186] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 62.025831] vdf: detected capacity change from 0 to 4294967296 [ 62.131948] libphy: Fixed MDIO Bus: probed [ 62.186614] usbcore: registered new interface driver usbserial_generic [ 62.222893] usbserial: USB Serial support registered for generic [ 62.231985] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 62.262841] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 62.284882] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 62.299673] mousedev: PS/2 mouse device common for all mice [ 62.327212] rtc_cmos 00:05: RTC can wake from S4 [ 62.349052] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 62.352857] rtc_cmos 00:05: registered as rtc0 [ 62.391163] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 62.393881] intel_pstate: CPU model not supported [ 62.424189] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 62.457945] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 62.460756] hid: raw HID events driver (C) Jiri Kosina [ 62.488176] usbcore: registered new interface driver usbhid [ 62.490949] usbhid: USB HID core driver [ 62.493181] drop_monitor: Initializing network drop monitor service [ 62.496454] Initializing XFRM netlink socket [ 62.513111] NET: Registered protocol family 10 [ 62.558232] Segment Routing with IPv6 [ 62.572209] NET: Registered protocol family 17 [ 62.590940] mpls_gso: MPLS GSO support [ 62.632852] RAS: Correctable Errors collector initialized. [ 62.634529] AVX version of gcm_enc/dec engaged. [ 62.666050] AES CTR mode by8 optimization enabled [ 63.685712] sched_clock: Marking stable (63685641829, 0)->(67847141100, -4161499271) [ 63.698799] registered taskstats version 1 [ 63.717520] Loading compiled-in X.509 certificates [ 63.720295] zswap: loaded using pool lzo/zbud [ 64.117644] Key type big_key registered [ 64.275659] Key type encrypted registered [ 64.289460] ima: No TPM chip found, activating TPM-bypass! [ 64.302261] ima: Allocated hash algorithm: sha1 [ 64.311945] ima: No architecture policies found [ 64.324956] evm: Initialising EVM extended attributes: [ 64.347958] evm: security.selinux [ 64.356826] evm: security.ima [ 64.370324] evm: security.capability [ 64.384711] evm: HMAC attrs: 0x1 [ 64.454379] rtc_cmos 00:05: setting system clock to 2025-04-01 07:34:22 UTC (1743492862) [ 64.639950] debug: unmapping init [mem 0xffffffff97a03000-0xffffffff97bfffff] [ 64.660835] debug: unmapping init [mem 0xffffffff96071000-0xffffffff96502fff] [ 64.672833] Write protecting the kernel read-only data: 30720k [ 64.683721] debug: unmapping init [mem 0xffffffff94603000-0xffffffff947fffff] [ 64.705195] debug: unmapping init [mem 0xffffffff94f2f000-0xffffffff94ffffff] [ 65.592748] systemd[1]: systemd 239 (239-82.el8_10.3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 65.645524] systemd[1]: Detected virtualization kvm. [ 65.654909] systemd[1]: Detected architecture x86-64. [ 65.669515] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 65.857067] systemd[1]: No hostname configured. [ 65.859068] systemd[1]: Set hostname to . [ 65.862141] random: systemd: uninitialized urandom read (16 bytes read) [ 65.876661] systemd[1]: Initializing machine ID from random generator. [ 66.390914] random: ln: uninitialized urandom read (6 bytes read) [ 67.871362] random: systemd: uninitialized urandom read (16 bytes read) [ 67.886442] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 67.939897] random: systemd: uninitialized urandom read (16 bytes read) [ 67.960323] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 67.996627] random: systemd: uninitialized urandom read (16 bytes read) [ 68.013615] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ 69.055637] systemd-tmpfile (234) used greatest stack depth: 13184 bytes left Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ 70.627229] systemd[1]: Started Setup Virtual Console. [ OK ] Started Setup Virtual Console. [ 70.739411] systemd[1]: Started Create list of required static device nodes for the current kernel. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ 70.866144] systemd[1]: Started Apply Kernel Variables. [ OK ] Started Apply Kernel Variables. [ 70.950243] systemd[1]: Starting Create Static Device Nodes in /dev... Starting Create Static Device Nodes in /dev... [ 71.113354] systemd[1]: Starting dracut cmdline hook... Starting dracut cmdline hook... [ 71.368332] systemd[1]: Started Create Static Device Nodes in /dev. [ OK ] Started Create Static Device Nodes in /dev. [ 71.842621] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 75.674382] device-mapper: uevent: version 1.0.3 [ 75.689439] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 86.749532] random: fast init done [ 87.679312] virtio_net virtio0 ens2: renamed from eth0 [ 94.648460] random: crng init done [ 94.650048] random: 5 urandom warning(s) missed due to ratelimiting [ 94.878035] scsi host0: ata_piix [ 95.119966] scsi host1: ata_piix [ 95.144852] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 95.166223] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 96.168056] systemd-udevd (442) used greatest stack depth: 12536 bytes left [ 97.994827] ip (527) used greatest stack depth: 11496 bytes left [ 103.929232] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 111.918419] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 124.853616] printk: systemd: 19 output lines suppressed due to ratelimiting [ 127.753290] SELinux: Disabled at runtime. [ 128.190129] systemd[1]: systemd 239 (239-82.el8_10.3) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 128.291593] systemd[1]: Detected virtualization kvm. [ 128.294884] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 133.876836] systemd[1]: initrd-switch-root.service: Succeeded. [ 133.930356] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 133.959190] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 133.984409] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 133.996837] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 134.288499] systemd[1]: Starting Journal Service... Starting Journal Service... [ 134.351729] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ 134.518391] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 134.785179] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ 134.885074] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ 136.425180] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 141.553169] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 147.428944] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 148.460371] input: PC Speaker as /devices/platform/pcspkr/input/input5 [* ] A start job is running for Configur…only root support (17s / no limit) [** ] A start job is running for Configur…only root support (18s / no limit) [*** ] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (19s / no limit) [ *** ] A start job is running for Configur…only root support (20s / no limit) [ ***] A start job is running for Configur…only root support (20s / no limit) [ **] A start job is running for Configur…only root support (21s / no limit) [ *] A start job is running for Configur…only root support (22s / no limit) [ **] A start job is running for Configur…only root support (22s / no limit)[ 156.127446] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ ***] A start job is running for Configur…only root support (23s / no limit)[ 156.563816] EDAC sbridge: Ver: 1.1.2 [ *** ] A start job is running for Configur…only root support (23s / no limit) [ *** ] A start job is running for Configur…only root support (24s / no limit) [*** ] A start job is running for Configur…only root support (27s / no limit) [** ] A start job is running for Configur…only root support (28s / no limit) [* ] A start job is running for Configur…only root support (28s / no limit) [** ] A start job is running for Configur…only root support (29s / no limit) [*** ] A start job is running for Configur…only root support (29s / no limit) [ *** ] A start job is running for Configur…only root support (31s / no limit) [ *** ] A start job is running for Configur…only root support (31s / no limit) [ ***] A start job is running for Configur…only root support (32s / no limit) [ **] A start job is running for Configur…only root support (32s / no limit) [ *] A start job is running for Configur…only root support (33s / no limit) [ **] A start job is running for Configur…only root support (33s / no limit) [ ***] A start job is running for Configur…only root support (34s / no limit) [ *** ] A start job is running for Configur…only root support (34s / no limit) [ *** ] A start job is running for Configur…only root support (35s / no limit) [*** ] A start job is running for Configur…only root support (35s / no limit) [** ] A start job is running for Configur…only root support (36s / no limit) [* ] A start job is running for Configur…only root support (36s / no limit) [** ] A start job is running for Configur…only root support (37s / no limit) [*** ] A start job is running for Configur…only root support (37s / no limit) [ *** ] A start job is running for Configur…only root support (38s / no limit) [ *** ] A start job is running for Configur…only root support (38s / no limit) [ ***] A start job is running for Configur…only root support (39s / no limit) [ **] A start job is running for Configur…only root support (40s / no limit)[ 173.362381] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (40s / no limit) [ **] A start job is running for Configur…only root support (41s / no limit) [ ***] A start job is running for Configur…only root support (41s / no limit)[ 174.943225] NFS: Registering the id_resolver key type [ 174.951479] Key type id_resolver registered [ 174.957213] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (42s / no limit) [ *** ] A start job is running for Configur…only root support (42s / no limit)[ 176.171579] mount.nfs (977) used greatest stack depth: 10760 bytes left [*** ] A start job is running for Configur…only root support (43s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg651-server login: [ 293.739582] libcfs: loading out-of-tree module taints kernel. [ 293.779732] Key type ._llcrypt registered [ 293.782623] Key type .llcrypt registered [ 294.104467] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 331.998976] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 338.965867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 339.044271] alg: No test for adler32 (adler32-zlib) [ 341.484379] Lustre: Lustre: Build Version: 2.16.52_73_g6bb624e [ 342.888196] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 342.892566] LNet: Accept secure, port 988 [ 344.904722] Key type lgssc registered [ 348.716538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 383.868321] ZFS: Loaded module v2.3.0-1, ZFS pool version 5000, ZFS filesystem version 5 [ 383.887580] modprobe (4345) used greatest stack depth: 5648 bytes left [ 390.203776] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 413.051530] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 428.603637] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 445.387835] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 486.985118] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 520.674112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 520.858843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 520.894918] ------------[ cut here ]------------ [ 520.897148] DEBUG_LOCKS_WARN_ON(!lockdep_enabled()) [ 520.897178] WARNING: CPU: 3 PID: 6534 at kernel/locking/lockdep.c:4713 lockdep_init_map_type+0x29d/0x410 [ 520.902762] Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [ 520.925862] CPU: 3 PID: 6534 Comm: mount.lustre Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #7 [ 520.935614] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 [ 520.945958] RIP: 0010:lockdep_init_map_type+0x29d/0x410 [ 520.947728] Code: c0 0f 85 db fe ff ff 48 c7 c6 66 64 d9 94 48 c7 c7 a7 4e d7 94 48 83 05 f0 df 25 03 01 e8 27 14 f5 ff 48 83 05 eb df 25 03 01 <0f> 0b 48 83 05 e9 df 25 03 01 48 83 05 e9 df 25 03 01 e9 a1 fe ff [ 520.965918] RSP: 0018:ffffc24842347748 EFLAGS: 00010202 [ 520.968246] RAX: 0000000000000000 RBX: ffff9e964011a370 RCX: 0000000000000001 [ 520.970735] RDX: 0000000000000001 RSI: 00000000ffff7fff RDI: ffff9e9641fde800 [ 520.972552] RBP: ffffffffc15f87c0 R08: 0000000000000000 R09: c0000000ffff7fff [ 520.974564] R10: 0000000000000001 R11: ffffc24842347538 R12: 0000000000000002 [ 520.977008] R13: ffff9e960774c000 R14: 0000000000000000 R15: 0000000000000001 [ 520.979750] FS: 00007f9dfdb08b40(0000) GS:ffff9e9641e00000(0000) knlGS:0000000000000000 [ 520.982075] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 520.983840] CR2: 00007f3fcffb5000 CR3: 0000000112d27006 CR4: 0000000000170ee0 [ 520.986145] Call Trace: [ 520.986880] ? show_regs.cold.9+0x22/0x2f [ 520.988067] ? __warn+0xc8/0x150 [ 520.989095] ? lockdep_init_map_type+0x29d/0x410 [ 520.990704] ? report_bug+0x113/0x140 [ 520.991958] ? do_error_trap+0xb6/0x130 [ 520.993323] ? do_invalid_op+0x46/0x60 [ 520.994649] ? lockdep_init_map_type+0x29d/0x410 [ 520.997319] ? invalid_op+0x14/0x20 [ 521.002126] ? lockdep_init_map_type+0x29d/0x410 [ 521.003249] ? lockdep_init_map_type+0x295/0x410 [ 521.004452] ldiskfs_enable_quotas+0x1b9/0x4a0 [ldiskfs] [ 521.011676] ldiskfs_fill_super+0x3a56/0x43c0 [ldiskfs] [ 521.018544] ? ldiskfs_calculate_overhead+0x670/0x670 [ldiskfs] [ 521.026646] ? mount_bdev+0x226/0x270 [ 521.032876] mount_bdev+0x226/0x270 [ 521.034125] ldiskfs_mount+0x19/0x30 [ldiskfs] [ 521.046074] legacy_get_tree+0x35/0x90 [ 521.047414] vfs_get_tree+0x2a/0x140 [ 521.052028] fc_mount+0x16/0x60 [ 521.059687] vfs_kern_mount+0x91/0x100 [ 521.060994] osd_mount+0x5c4/0x1080 [osd_ldiskfs] [ 521.068274] osd_device_init0+0x2e1/0xc20 [osd_ldiskfs] [ 521.078857] osd_device_alloc+0x22a/0x290 [osd_ldiskfs] [ 521.085191] obd_setup+0x196/0x430 [obdclass] [ 521.089280] class_setup+0x6f5/0x9f0 [obdclass] [ 521.092203] class_process_config+0x1658/0x2b60 [obdclass] [ 521.103697] do_lcfg+0x376/0x740 [obdclass] [ 521.108758] lustre_start_simple+0x8f/0x220 [obdclass] [ 521.125957] osd_start+0x6aa/0xb60 [ptlrpc] [ 521.127476] ? server_name2index+0x79/0xe0 [obdclass] [ 521.132831] ? lsi_prepare+0x2e7/0x690 [ptlrpc] [ 521.137471] server_fill_super+0x99/0x1190 [ptlrpc] [ 521.141557] ? obd_zombie_barrier+0x63/0x120 [obdclass] [ 521.149718] ? debug_mutex_init+0x43/0x60 [ 521.160637] lustre_fill_super+0x4a6/0x5e0 [lustre] [ 521.162417] ? lustre_mount+0x30/0x30 [lustre] [ 521.166807] mount_nodev+0x56/0xf0 [ 521.169978] lustre_mount+0x1c/0x30 [lustre] [ 521.173209] legacy_get_tree+0x35/0x90 [ 521.174400] vfs_get_tree+0x2a/0x140 [ 521.181879] do_mount+0xd84/0x1190 [ 521.188666] ksys_mount+0x11d/0x150 [ 521.189772] __x64_sys_mount+0x29/0x40 [ 521.202415] do_syscall_64+0xc1/0x450 [ 521.203569] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 521.205299] RIP: 0033:0x7f9df9c3cdbe [ 521.215014] Code: 48 8b 0d cd 60 39 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 9a 60 39 00 f7 d8 64 89 01 48 [ 521.234700] RSP: 002b:00007fff1388e888 EFLAGS: 00000286 ORIG_RAX: 00000000000000a5 [ 521.243964] RAX: ffffffffffffffda RBX: 0000000000430cf6 RCX: 00007f9df9c3cdbe [ 521.251898] RDX: 0000000000430cf6 RSI: 00007fff13894f30 RDI: 0000000001dcc940 [ 521.273681] RBP: 00007fff13894f30 R08: 0000000001dcc960 R09: 0000000001dcc010 [ 521.280040] R10: 0000000001000000 R11: 0000000000000286 R12: 0000000000000000 [ 521.282484] R13: 0000000000654920 R14: 00000000fffffff5 R15: 00000000fffffff5 [ 521.294835] ---[ end trace 911b99b5637a1c00 ]--- [ 521.304707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.975845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 523.062844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 523.187907] Lustre: lustre-MDT0000: new disk, initializing [ 523.349694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.399359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 529.929164] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 549.783939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 549.959433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 550.175968] Lustre: Modifying parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 550.229093] Lustre: 7466:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 550.263551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 550.266937] Lustre: Skipped 1 previous similar message [ 550.366839] Lustre: lustre-MDT0001: new disk, initializing [ 550.488204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 550.553106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 550.569537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 556.506437] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 563.575111] Lustre: Modifying parameter general.debug_raw_pointers in log params [ 579.420071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 579.650970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 580.220444] Lustre: 8424:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 580.267543] Lustre: lustre-OST0000: new disk, initializing [ 580.287653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 580.482429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 588.904088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 588.921089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 589.251189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 589.300519] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 610.244916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 610.436604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 610.802078] Lustre: 9441:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 610.870931] Lustre: lustre-OST0001: new disk, initializing [ 610.895987] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 611.040224] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 618.600443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 618.616556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 618.876198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 621.490711] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 639.547199] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 651.939959] Lustre: Setting parameter general.lod.*.mdt_hash in log params [ 667.828929] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing check_logdir /tmp/testlogs/ [ 677.832414] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing yml_node [ 693.023157] Lustre: DEBUG MARKER: Client: 2.16.52.73 [ 698.169575] Lustre: DEBUG MARKER: MDS: 2.16.52.73 [ 704.581195] Lustre: DEBUG MARKER: OSS: 2.16.52.73 [ 708.366737] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Apr 1 03:45:02 EDT 2025 [ 741.160578] Lustre: DEBUG MARKER: excepting tests: 110f 131b 59 36 [ 746.092615] Lustre: DEBUG MARKER: === replay-single: start setup 03:45:40 (1743493540) === [ 754.307482] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing check_config_client /mnt/lustre [ 790.822455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 798.760815] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 805.257086] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 817.849530] Lustre: DEBUG MARKER: === replay-single: finish setup 03:46:51 (1743493611) === [ 822.564249] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 03:46:56 (1743493616) [ 834.941580] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 838.119296] Lustre: Failing over lustre-MDT0000 [ 838.520766] Lustre: server umount lustre-MDT0000 complete [ 839.148534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.172521] Lustre: Skipped 1 previous similar message [ 844.258719] LustreError: 6562:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 844.274669] LustreError: 6562:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 9 previous similar messages [ 845.071622] LustreError: 6558:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.377567] LustreError: 6562:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 849.417340] LustreError: 6562:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 3 previous similar messages [ 854.511262] LustreError: 6562:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 854.536685] LustreError: 6562:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 4 previous similar messages [ 855.521294] Lustre: 3682:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743493637/real 1743493637] req@ffff9e963a2b1740 x1828185062938496/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743493653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 855.575575] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 860.651640] LustreError: 6556:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 860.659959] LustreError: 6556:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 8 previous similar messages [ 870.683804] LustreError: 8910:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.721543] LustreError: 8910:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 6 previous similar messages [ 872.817290] LDISKFS-fs (dm-0): recovery complete [ 872.831979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.773126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.883903] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 880.922940] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 885.768472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 885.868263] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 888.030732] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 901.886506] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 906.402939] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 926.253561] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 03:48:40 (1743493720) [ 931.298127] Lustre: Failing over lustre-OST0000 [ 931.523398] Lustre: server umount lustre-OST0000 complete [ 931.816171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.851816] Lustre: Skipped 2 previous similar messages [ 931.870367] LustreError: 8780:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 931.891950] LustreError: 8780:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 14 previous similar messages [ 958.098888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 958.165210] Lustre: 15865:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 958.383459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.409839] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 960.059864] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 969.717359] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 976.219805] Lustre: lustre-OST0000: Recovery over after 0:16, of 3 clients 3 recovered and 0 were evicted. [ 976.221032] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 976.242104] Lustre: Skipped 3 previous similar messages [ 984.363925] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 988.240524] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 1009.304501] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 03:50:03 (1743493803) [ 1022.854272] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1027.273182] Lustre: Failing over lustre-MDT0000 [ 1027.874891] Lustre: server umount lustre-MDT0000 complete [ 1029.601667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1029.616208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.685513] Lustre: Skipped 1 previous similar message [ 1029.715921] LustreError: 6562:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1029.724926] LustreError: 6562:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 10 previous similar messages [ 1032.163623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.182493] Lustre: Skipped 2 previous similar messages [ 1048.529703] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743493830/real 1743493830] req@ffff9e963a2b5680 x1828185063034880/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743493846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1048.547873] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1060.721726] LDISKFS-fs (dm-0): recovery complete [ 1060.724618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.545086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.607369] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1078.764862] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1078.786335] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 1078.789578] Lustre: Skipped 1 previous similar message [ 1080.001934] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1084.240029] Lustre: lustre-MDT0000: Denying connection for new client 003f424d-53b1-47ef-bf5e-c77599647c7b (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 1089.807220] Lustre: lustre-MDT0000: Denying connection for new client 003f424d-53b1-47ef-bf5e-c77599647c7b (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 1089.831424] Lustre: Skipped 1 previous similar message [ 1094.926531] Lustre: lustre-MDT0000: Denying connection for new client 003f424d-53b1-47ef-bf5e-c77599647c7b (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 1100.047412] Lustre: lustre-MDT0000: Denying connection for new client 003f424d-53b1-47ef-bf5e-c77599647c7b (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:47 [ 1105.167880] Lustre: lustre-MDT0000: Denying connection for new client 003f424d-53b1-47ef-bf5e-c77599647c7b (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:42 [ 1115.411275] Lustre: lustre-MDT0000: Denying connection for new client 003f424d-53b1-47ef-bf5e-c77599647c7b (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 1115.438981] Lustre: Skipped 1 previous similar message [ 1135.886254] Lustre: lustre-MDT0000: Denying connection for new client 003f424d-53b1-47ef-bf5e-c77599647c7b (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:12 [ 1135.903799] Lustre: Skipped 3 previous similar messages [ 1148.003419] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1148.023296] Lustre: 17802:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 50e06636-072e-49b7-b9c0-cbffa595444c@ [ 1148.034446] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1148.064312] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 1148.067561] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 1148.090982] Lustre: Skipped 2 previous similar messages [ 1148.128701] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 1148.136630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 1171.292516] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 03:52:45 (1743493965) [ 1185.038384] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1188.903064] Lustre: Failing over lustre-MDT0000 [ 1189.263906] Lustre: server umount lustre-MDT0000 complete [ 1189.346531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1189.357510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.387687] LustreError: 7825:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1189.433190] LustreError: 7825:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 40 previous similar messages [ 1207.776228] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743493989/real 1743493989] req@ffff9e963a0e4b00 x1828185063103744/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743494005 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1207.832507] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1220.649365] LDISKFS-fs (dm-0): recovery complete [ 1220.668649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1232.357041] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9514944b00 x1828185063116544/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1232.651308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.723619] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1237.994540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1238.014086] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 1238.995228] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1243.088393] Lustre: lustre-MDT0000: Denying connection for new client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 1243.106791] Lustre: Skipped 2 previous similar messages [ 1307.001335] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 1307.007789] Lustre: 19513:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 003f424d-53b1-47ef-bf5e-c77599647c7b@ [ 1307.045293] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1307.074190] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 1307.075811] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 1307.105844] Lustre: Skipped 2 previous similar messages [ 1307.158297] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 1307.170038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 1329.363911] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 03:55:23 (1743494123) [ 1343.137448] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1346.773858] Lustre: Failing over lustre-MDT0000 [ 1347.184344] Lustre: server umount lustre-MDT0000 complete [ 1348.067819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1348.074757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.083311] Lustre: Skipped 3 previous similar messages [ 1348.087472] LustreError: 7825:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.103387] LustreError: 7825:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 40 previous similar messages [ 1366.994832] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743494148/real 1743494148] req@ffff9e96073c39c0 x1828185063171328/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743494164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1367.020348] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1380.998671] LDISKFS-fs (dm-0): recovery complete [ 1381.008871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1392.618261] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb427bd [ 1392.630125] Lustre: MGC192.168.206.151@tcp: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 1393.241739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.326281] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1393.781284] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1398.320221] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1398.394782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 1398.395674] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 1400.338376] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1414.518687] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1418.951819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1437.478573] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 03:57:12 (1743494232) [ 1451.092249] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1454.539569] Lustre: Failing over lustre-MDT0000 [ 1454.562024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1454.566370] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1454.589311] Lustre: Skipped 4 previous similar messages [ 1454.593267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1454.901125] Lustre: server umount lustre-MDT0000 complete [ 1475.043285] Lustre: 3682:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743494257/real 1743494257] req@ffff9e95137939c0 x1828185063223040/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743494273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1475.091928] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1487.691901] LDISKFS-fs (dm-0): recovery complete [ 1487.700415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.649534] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9514942880 x1828185063235072/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1501.234078] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1501.470534] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1506.308742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 1506.316380] Lustre: Skipped 4 previous similar messages [ 1506.477811] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1506.537143] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 1506.544169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1508.457994] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1522.530944] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1527.430732] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1547.182254] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 03:59:01 (1743494341) [ 1558.964226] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1562.154821] Lustre: Failing over lustre-MDT0000 [ 1562.472775] Lustre: server umount lustre-MDT0000 complete [ 1562.596872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.600347] Lustre: Skipped 2 previous similar messages [ 1584.114766] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743494365/real 1743494365] req@ffff9e962fd7c540 x1828185063275392/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743494381 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1584.142379] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1593.840354] LDISKFS-fs (dm-0): recovery complete [ 1593.845897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1594.353841] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb431c6 [ 1594.369117] Lustre: MGC192.168.206.151@tcp: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 1594.386162] Lustre: Skipped 3 previous similar messages [ 1594.668919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.672027] Lustre: Skipped 1 previous similar message [ 1594.754798] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1598.742963] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1600.150262] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 1600.224570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 1600.257682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 1600.776662] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1615.211158] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1619.594954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1638.688203] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 04:00:32 (1743494432) [ 1651.014996] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1654.343954] Lustre: Failing over lustre-MDT0000 [ 1654.821166] Lustre: server umount lustre-MDT0000 complete [ 1656.290964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.303570] Lustre: Skipped 3 previous similar messages [ 1656.324741] LustreError: 15213:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.331928] LustreError: 15213:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 137 previous similar messages [ 1672.673204] Lustre: 3680:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743494454/real 1743494454] req@ffff9e9515b93f80 x1828185063321984/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743494470 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1672.729024] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1687.016552] LDISKFS-fs (dm-0): recovery complete [ 1687.019329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.253993] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb436f8 [ 1697.886253] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1700.114184] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1703.052721] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1703.117827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1703.117827] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 1704.404375] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1717.944862] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1722.896139] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1740.921207] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 04:02:15 (1743494535) [ 1753.971272] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1757.696889] Lustre: Failing over lustre-MDT0000 [ 1758.177791] Lustre: server umount lustre-MDT0000 complete [ 1775.590759] Lustre: 3681:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743494557/real 1743494557] req@ffff9e9626134b00 x1828185063374208/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743494573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1775.638607] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1789.011040] LDISKFS-fs (dm-0): recovery complete [ 1789.017390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.167097] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb43c0e [ 1800.195035] Lustre: MGC192.168.206.151@tcp: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 1800.198548] Lustre: Skipped 9 previous similar messages [ 1800.747755] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1802.519483] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1805.878125] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1805.921589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1805.927968] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 1807.311771] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1821.832918] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1825.778339] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1844.474538] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 04:03:58 (1743494638) [ 1846.161478] Lustre: *** cfs_fail_loc=13b, val=315*** [ 1846.163628] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 1846.168689] LustreError: 6560:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e9608e20040 x1828185024158976/t38654705666(0) o35->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:335/0 lens 392/456 e 0 to 0 dl 1743494655 ref 1 fl Interpret:/200/0 rc 0/0 job:'openfile.0' uid:0 gid:0 [ 1860.398663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1862.498871] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnecting [ 1862.523712] Lustre: 6560:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e9515b922c0 x1828185024158976/t38654705666(0) o35->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:351/0 lens 392/456 e 0 to 0 dl 1743494671 ref 1 fl Interpret:/202/0 rc 0/0 job:'openfile.0' uid:0 gid:0 [ 1863.842595] Lustre: Failing over lustre-MDT0000 [ 1864.393709] Lustre: server umount lustre-MDT0000 complete [ 1867.240715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1867.246452] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.282865] Lustre: Skipped 10 previous similar messages [ 1883.616951] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743494665/real 1743494665] req@ffff9e963a35e200 x1828185063426944/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743494681 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 1883.665303] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1897.966957] LDISKFS-fs (dm-0): recovery complete [ 1897.969752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.716206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.731320] Lustre: Skipped 2 previous similar messages [ 1913.968157] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 1913.969716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1915.806702] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1931.047080] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1935.769040] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1955.941396] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 04:05:49 (1743494749) [ 1968.751727] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1971.889391] Lustre: Failing over lustre-MDT0000 [ 1972.022387] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1972.027965] Lustre: Skipped 3 previous similar messages [ 1972.264293] Lustre: server umount lustre-MDT0000 complete [ 1975.264855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2004.636408] LDISKFS-fs (dm-0): recovery complete [ 2004.641704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2016.227271] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9515b97340 x1828185063491072/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2016.870889] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2016.885671] Lustre: Skipped 1 previous similar message [ 2018.067993] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2018.085937] Lustre: Skipped 1 previous similar message [ 2022.034721] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2022.049956] Lustre: Skipped 1 previous similar message [ 2022.119199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 2022.121694] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2023.780594] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2037.336594] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2041.160134] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2061.222236] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 04:07:34 (1743494854) [ 2074.249159] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2075.998026] Lustre: *** cfs_fail_loc=114, val=0*** [ 2080.618744] Lustre: Failing over lustre-MDT0000 [ 2081.030902] Lustre: server umount lustre-MDT0000 complete [ 2099.680184] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743494881/real 1743494881] req@ffff9e960774ae40 x1828185063532032/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743494897 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2099.706950] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2099.710635] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.738170] LustreError: Skipped 1 previous similar message [ 2112.394761] LDISKFS-fs (dm-0): recovery complete [ 2112.405904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2124.260639] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e963a3f9740 x1828185063544704/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2129.914350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 2129.935566] Lustre: Skipped 12 previous similar messages [ 2130.020402] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 2130.023576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2131.276451] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2145.585889] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2149.932852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2168.635963] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 04:09:22 (1743494962) [ 2181.009375] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2182.749316] Lustre: *** cfs_fail_loc=128, val=0*** [ 2187.111616] Lustre: Failing over lustre-MDT0000 [ 2187.481922] Lustre: server umount lustre-MDT0000 complete [ 2191.330795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.332023] LustreError: 6556:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.339625] Lustre: Skipped 11 previous similar messages [ 2191.369938] LustreError: 6556:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 246 previous similar messages [ 2219.132342] LDISKFS-fs (dm-0): recovery complete [ 2219.142633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.313301] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e962fd79d00 x1828185063596800/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2239.121382] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 2239.121562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2239.921600] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2253.066367] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2256.622766] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2273.791133] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 04:11:08 (1743495068) [ 2285.844337] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2288.974633] Lustre: Failing over lustre-MDT0000 [ 2289.593806] Lustre: server umount lustre-MDT0000 complete [ 2321.405243] LDISKFS-fs (dm-0): recovery complete [ 2321.409151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.674639] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2331.690720] Lustre: Skipped 2 previous similar messages [ 2333.196580] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2333.212126] Lustre: Skipped 2 previous similar messages [ 2337.264216] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2337.271371] Lustre: Skipped 2 previous similar messages [ 2337.318111] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 2337.322079] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 2338.509835] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2350.799502] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2354.775107] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2372.660236] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 04:12:47 (1743495167) [ 2385.739227] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2389.179869] Lustre: Failing over lustre-MDT0000 [ 2389.687329] Lustre: server umount lustre-MDT0000 complete [ 2409.444411] Lustre: 3680:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743495191/real 1743495191] req@ffff9e9626135c40 x1828185063687680/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743495207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2409.489855] Lustre: 3680:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2409.501264] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.514434] LustreError: Skipped 2 previous similar messages [ 2420.093873] LDISKFS-fs (dm-0): recovery complete [ 2420.096828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.021691] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb46192 [ 2434.569279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2434.584508] Lustre: Skipped 4 previous similar messages [ 2439.971069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 2439.973234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 2441.547396] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2455.842515] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2460.290970] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2479.287293] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 04:14:33 (1743495273) [ 2491.800440] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2503.084754] Lustre: Failing over lustre-MDT0000 [ 2503.408956] Lustre: server umount lustre-MDT0000 complete [ 2533.528588] LDISKFS-fs (dm-0): recovery complete [ 2533.531693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.168764] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2558.352215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 2558.353979] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 2567.151035] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2571.122302] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2609.716258] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 04:16:44 (1743495404) [ 2621.679759] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2625.075740] Lustre: Failing over lustre-MDT0000 [ 2625.362479] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 2625.549148] Lustre: server umount lustre-MDT0000 complete [ 2656.759738] LDISKFS-fs (dm-0): recovery complete [ 2656.761782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2673.660552] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 2673.674621] Lustre: Skipped 20 previous similar messages [ 2673.842839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 2673.844644] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 2674.848116] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2687.737563] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2691.235387] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2711.286558] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 04:18:25 (1743495505) [ 2723.547851] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2727.145616] Lustre: Failing over lustre-MDT0000 [ 2727.469566] Lustre: server umount lustre-MDT0000 complete [ 2729.967521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.973248] Lustre: Skipped 18 previous similar messages [ 2757.943180] LDISKFS-fs (dm-0): recovery complete [ 2757.948600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.688705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 2777.690798] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 2777.962465] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2790.175784] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2794.775768] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2813.956192] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 04:20:08 (1743495608) [ 2826.976141] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2830.682389] Lustre: Failing over lustre-MDT0000 [ 2831.353774] Lustre: server umount lustre-MDT0000 complete [ 2833.895676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2833.898885] LustreError: 18156:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.928335] LustreError: 18156:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 293 previous similar messages [ 2863.149113] LDISKFS-fs (dm-0): recovery complete [ 2863.153784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.333223] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2875.343253] Lustre: Skipped 4 previous similar messages [ 2876.173778] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2876.191532] Lustre: Skipped 4 previous similar messages [ 2880.596252] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2880.599808] Lustre: Skipped 4 previous similar messages [ 2880.664243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 2880.665913] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 2881.889880] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2895.615810] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2899.769527] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2917.891511] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 04:21:52 (1743495712) [ 2930.338803] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2933.420471] Lustre: Failing over lustre-MDT0000 [ 2933.753348] Lustre: server umount lustre-MDT0000 complete [ 2953.184127] Lustre: 3680:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743495734/real 1743495734] req@ffff9e963a351740 x1828185064018944/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743495750 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2953.231692] Lustre: 3680:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 2953.247289] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.267259] LustreError: Skipped 4 previous similar messages [ 2963.934379] LDISKFS-fs (dm-0): recovery complete [ 2963.940094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2978.786279] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9514dc5c40 x1828185064031360/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 2984.631845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 2984.637173] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 2986.240702] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3001.342437] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3005.029408] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3023.697732] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 04:23:37 (1743495817) [ 3036.934439] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3040.870199] Lustre: Failing over lustre-MDT0000 [ 3041.249719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3041.269370] Lustre: server umount lustre-MDT0000 complete [ 3074.040465] LDISKFS-fs (dm-0): recovery complete [ 3074.046215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.275514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.287975] Lustre: Skipped 5 previous similar messages [ 3092.656730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 3092.661205] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 3094.159328] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3107.869549] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3112.313531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3131.687536] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 04:25:26 (1743495926) [ 3145.019975] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3148.942583] Lustre: Failing over lustre-MDT0000 [ 3149.293299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3149.467240] Lustre: server umount lustre-MDT0000 complete [ 3182.144602] LDISKFS-fs (dm-0): recovery complete [ 3182.147167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.849542] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e95134639c0 x1828185064136832/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 3200.700715] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 3200.704211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 3201.770871] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3215.312586] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3219.109925] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3236.977899] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 04:27:11 (1743496031) [ 3248.960501] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3252.057826] Lustre: Failing over lustre-MDT0000 [ 3252.206115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3252.222621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.406930] Lustre: server umount lustre-MDT0000 complete [ 3282.196874] LDISKFS-fs (dm-0): recovery complete [ 3282.200935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.400521] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9506a8f900 x1828185064184448/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 3288.051421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 3288.064613] Lustre: Skipped 23 previous similar messages [ 3288.275291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 3288.277401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 3288.856277] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3301.485386] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3305.267653] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3323.908768] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 04:28:38 (1743496118) [ 3335.922535] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3339.234295] Lustre: Failing over lustre-MDT0000 [ 3339.251881] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3339.261805] Lustre: Skipped 21 previous similar messages [ 3339.271423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3339.288056] Lustre: Skipped 3 previous similar messages [ 3339.573563] Lustre: server umount lustre-MDT0000 complete [ 3369.600398] LDISKFS-fs (dm-0): recovery complete [ 3369.623329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.269327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 3376.282677] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 3377.107971] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3390.272825] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3394.572149] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3413.836893] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 04:30:07 (1743496207) [ 3428.577230] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3432.034847] Lustre: Failing over lustre-MDT0000 [ 3432.493107] Lustre: server umount lustre-MDT0000 complete [ 3437.538328] LustreError: 15213:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3437.569992] LustreError: 15213:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 254 previous similar messages [ 3463.854392] LDISKFS-fs (dm-0): recovery complete [ 3463.865234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.499364] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9515519d00 x1828185064284416/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 3479.094633] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3479.096850] Lustre: Skipped 5 previous similar messages [ 3479.821472] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3479.824041] Lustre: Skipped 5 previous similar messages [ 3484.178540] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3484.189973] Lustre: Skipped 5 previous similar messages [ 3484.278774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 3484.318870] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 3486.174897] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3500.190571] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3504.209422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3521.734984] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 04:31:56 (1743496316) [ 3534.591893] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3537.874060] Lustre: Failing over lustre-MDT0000 [ 3538.245855] Lustre: server umount lustre-MDT0000 complete [ 3556.832110] Lustre: 3682:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743496338/real 1743496338] req@ffff9e963a134540 x1828185064323456/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743496354 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 3556.878586] Lustre: 3682:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 3556.897262] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3556.933181] LustreError: Skipped 5 previous similar messages [ 3568.329521] LDISKFS-fs (dm-0): recovery complete [ 3568.332366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.415917] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb52bc7 [ 3587.186294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:897) [ 3587.190301] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 3588.316282] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3603.031427] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3606.587948] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3627.391303] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 04:33:41 (1743496421) [ 3640.434364] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3643.749948] Lustre: Failing over lustre-MDT0000 [ 3644.131334] Lustre: server umount lustre-MDT0000 complete [ 3648.482281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3674.588309] LDISKFS-fs (dm-0): recovery complete [ 3674.599019] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3681.515026] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 3681.540769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 3682.853969] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3695.410769] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3699.241947] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3717.403499] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 04:35:11 (1743496511) [ 3728.937271] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3732.276144] Lustre: Failing over lustre-MDT0000 [ 3732.456710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3732.472137] Lustre: Skipped 3 previous similar messages [ 3732.742320] Lustre: server umount lustre-MDT0000 complete [ 3763.620167] LDISKFS-fs (dm-0): recovery complete [ 3763.622983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3778.415702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3778.426645] Lustre: Skipped 6 previous similar messages [ 3783.814742] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 3783.835313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 3784.665592] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3797.601387] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3801.581619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3820.081762] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 04:36:54 (1743496614) [ 3832.460336] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3835.551882] Lustre: Failing over lustre-MDT0000 [ 3836.049919] Lustre: server umount lustre-MDT0000 complete [ 3866.764416] LDISKFS-fs (dm-0): recovery complete [ 3866.766849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.953496] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e963a23e200 x1828185064487168/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 3887.912134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 3887.916402] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:993) [ 3888.902878] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3902.219152] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3906.788268] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3927.162729] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 04:38:40 (1743496720) [ 3940.579524] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3944.230807] Lustre: Failing over lustre-MDT0000 [ 3944.420751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3944.450290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3944.475741] Lustre: Skipped 23 previous similar messages [ 3944.503871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3944.961179] Lustre: server umount lustre-MDT0000 complete [ 3976.877413] LDISKFS-fs (dm-0): recovery complete [ 3976.879852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.644830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 3996.648273] Lustre: Skipped 28 previous similar messages [ 3996.966733] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 3996.969945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 3998.272216] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4011.912632] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4015.889186] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4034.715757] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 04:40:29 (1743496829) [ 4047.805910] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4051.090574] Lustre: Failing over lustre-MDT0000 [ 4051.394158] Lustre: server umount lustre-MDT0000 complete [ 4052.965429] LustreError: 6557:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4053.002096] LustreError: 6557:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 285 previous similar messages [ 4082.479642] LDISKFS-fs (dm-0): recovery complete [ 4082.494239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.570597] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4095.573174] Lustre: Skipped 5 previous similar messages [ 4096.182762] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4096.191746] Lustre: Skipped 5 previous similar messages [ 4100.846272] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4100.849150] Lustre: Skipped 5 previous similar messages [ 4100.921464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 4100.925687] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 4102.573419] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4116.056440] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4120.204950] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4138.538448] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 04:42:13 (1743496933) [ 4150.637708] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4154.065520] Lustre: Failing over lustre-MDT0000 [ 4154.504073] Lustre: server umount lustre-MDT0000 complete [ 4172.258950] Lustre: 3681:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743496954/real 1743496954] req@ffff9e9516d64b00 x1828185064633984/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743496970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 4172.316541] Lustre: 3681:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 4172.330441] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4172.344599] LustreError: Skipped 5 previous similar messages [ 4183.552889] LDISKFS-fs (dm-0): recovery complete [ 4183.558163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4197.861609] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb54ce4 [ 4203.716217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 4203.725741] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 4205.244697] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4217.702672] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4221.708804] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4238.423456] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 04:43:53 (1743497033) [ 4246.654073] Lustre: 71669:0:(genops.c:1678:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c154a474-28d3-40b8-9d00-0def439c1563 at adminstrative request [ 4256.469769] Lustre: Failing over lustre-MDT0000 [ 4256.501363] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 4256.894830] Lustre: server umount lustre-MDT0000 complete [ 4275.985487] LustreError: 41269:0:(ofd_io.c:776:ofd_preprw_write()) lustre-OST0000: BRW to missing obj 0x280000401:1090 [ 4282.175135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.409073] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e963f71ed80 x1828185064693760/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 4291.245781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 4291.253474] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 4291.323778] LustreError: 41271:0:(ofd_io.c:776:ofd_preprw_write()) lustre-OST0000: BRW to missing obj 0x280000401:1090 [ 4291.889391] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4304.593326] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4308.657842] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4317.037957] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 4331.844447] Lustre: DEBUG MARKER: before 3128, after 3128 [ 4345.849822] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 04:45:40 (1743497140) [ 4347.730702] Lustre: 73514:0:(genops.c:1678:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c154a474-28d3-40b8-9d00-0def439c1563 at adminstrative request [ 4364.261787] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 04:45:59 (1743497159) [ 4376.894480] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4379.834908] Lustre: Failing over lustre-MDT0000 [ 4380.152277] Lustre: server umount lustre-MDT0000 complete [ 4410.503619] LDISKFS-fs (dm-0): recovery complete [ 4410.506437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4424.162168] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e95141d6d80 x1828185064760832/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 4424.605345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4424.614038] Lustre: Skipped 5 previous similar messages [ 4430.043437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 4430.049378] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 4431.173371] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4443.494508] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4447.305553] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4464.166094] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 04:47:38 (1743497258) [ 4476.252588] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4479.455267] Lustre: Failing over lustre-MDT0000 [ 4480.089817] Lustre: server umount lustre-MDT0000 complete [ 4510.563339] LDISKFS-fs (dm-0): recovery complete [ 4510.573862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4528.741627] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 4528.743618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 4529.710054] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4543.110559] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4546.451957] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4563.535916] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 04:49:18 (1743497358) [ 4575.225507] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4578.232925] Lustre: Failing over lustre-MDT0000 [ 4578.555571] Lustre: server umount lustre-MDT0000 complete [ 4579.817160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.828569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4579.845642] Lustre: Skipped 24 previous similar messages [ 4606.844529] LDISKFS-fs (dm-0): recovery complete [ 4606.846941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.481231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 4627.493649] Lustre: Skipped 24 previous similar messages [ 4627.597639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 4627.602363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 4628.519191] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4641.451625] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4645.482459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4662.792501] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 04:50:57 (1743497457) [ 4673.834116] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4677.086473] Lustre: Failing over lustre-MDT0000 [ 4677.406659] Lustre: server umount lustre-MDT0000 complete [ 4678.641491] LustreError: 15213:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4678.658042] LustreError: 15213:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 270 previous similar messages [ 4706.799436] LDISKFS-fs (dm-0): recovery complete [ 4706.802310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.612294] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e950709a880 x1828185064910720/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 4720.625388] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb567a2 [ 4720.972531] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4720.977712] Lustre: Skipped 5 previous similar messages [ 4721.935794] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4721.951065] Lustre: Skipped 5 previous similar messages [ 4726.354175] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4726.378208] Lustre: Skipped 5 previous similar messages [ 4726.446510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 4726.447279] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 4726.448439] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4738.243246] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4741.458632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4756.718514] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 04:52:31 (1743497551) [ 4767.764263] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4770.750130] Lustre: Failing over lustre-MDT0000 [ 4771.125021] Lustre: server umount lustre-MDT0000 complete [ 4788.704978] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743497570/real 1743497570] req@ffff9e962fb2e200 x1828185064944896/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743497586 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 4788.748165] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 4788.773478] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4788.800926] LustreError: Skipped 5 previous similar messages [ 4799.892687] LDISKFS-fs (dm-0): recovery complete [ 4799.894914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4818.900382] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4819.006117] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 4819.008808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 4830.679632] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4834.126512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4850.264135] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 04:54:05 (1743497645) [ 4861.130309] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4864.338293] Lustre: Failing over lustre-MDT0000 [ 4864.722635] Lustre: server umount lustre-MDT0000 complete [ 4864.992777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4892.782677] LDISKFS-fs (dm-0): recovery complete [ 4892.787357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4896.747130] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb57292 [ 4902.015987] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4902.619523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 4902.626505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 4913.674355] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4917.235857] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4932.240211] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 04:55:27 (1743497727) [ 4942.773748] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4945.623669] Lustre: Failing over lustre-MDT0000 [ 4945.862342] Lustre: server umount lustre-MDT0000 complete [ 4971.019588] LDISKFS-fs (dm-0): recovery complete [ 4971.026672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.729620] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4979.842491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 4979.842711] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 4990.014851] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4993.304294] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5008.100164] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 04:56:42 (1743497802) [ 5019.184333] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5021.740623] Lustre: Failing over lustre-MDT0000 [ 5022.054971] Lustre: server umount lustre-MDT0000 complete [ 5048.256513] LDISKFS-fs (dm-0): recovery complete [ 5048.258911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5052.385500] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e960774b9c0 x1828185065087872/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 5052.428168] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 5052.696298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5052.701560] Lustre: Skipped 6 previous similar messages [ 5057.666860] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5058.154550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 5058.154565] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 5068.603470] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5071.960934] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5086.411535] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 04:58:01 (1743497881) [ 5096.452981] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5099.018384] Lustre: Failing over lustre-MDT0000 [ 5099.319815] Lustre: server umount lustre-MDT0000 complete [ 5126.487863] LDISKFS-fs (dm-0): recovery complete [ 5126.494886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.732157] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb58435 [ 5136.285613] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5136.474167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 5136.480900] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 5146.792723] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5150.233779] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5164.916728] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 04:59:19 (1743497959) [ 5175.934872] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5178.676293] Lustre: Failing over lustre-MDT0000 [ 5178.988351] Lustre: server umount lustre-MDT0000 complete [ 5182.444023] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5182.452389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5182.466621] Lustre: Skipped 27 previous similar messages [ 5182.478823] LustreError: Skipped 1 previous similar message [ 5205.578071] LDISKFS-fs (dm-0): recovery complete [ 5205.584042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5208.044551] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb58a24 [ 5213.479950] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5213.773199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 5213.773381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 5223.547599] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5226.319505] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5241.221159] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 05:00:36 (1743498036) [ 5250.799491] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5253.544191] Lustre: Failing over lustre-MDT0000 [ 5253.860515] Lustre: server umount lustre-MDT0000 complete [ 5278.538946] LDISKFS-fs (dm-0): recovery complete [ 5278.540699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.235483] LustreError: 6562:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.236828] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb58fe9 [ 5280.245113] LustreError: 6562:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 297 previous similar messages [ 5280.254720] Lustre: MGC192.168.206.151@tcp: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 5280.261032] Lustre: Skipped 35 previous similar messages [ 5285.022092] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5285.972749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 5285.989880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 5294.086690] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5297.096631] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5310.364138] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 05:01:45 (1743498105) [ 5311.868525] Lustre: 93828:0:(genops.c:1678:obd_export_evict_by_uuid()) lustre-MDT0000: evicting c154a474-28d3-40b8-9d00-0def439c1563 at adminstrative request [ 5327.450752] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 05:02:02 (1743498122) [ 5335.575070] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5337.869731] Lustre: Failing over lustre-MDT0000 [ 5338.090485] Lustre: server umount lustre-MDT0000 complete [ 5343.214928] LustreError: 86811:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_destroy_export: tot_granted 239680 != fo_tot_granted 2644032 [ 5343.224427] LustreError: 86811:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_destroy_export: tot_pending 0 != fo_tot_pending 2404352 [ 5343.244648] LustreError: 86811:0:(ofd_obd.c:483:ofd_destroy_export()) lustre-OST0000: cli c154a474-28d3-40b8-9d00-0def439c1563/ffff9e9517f2e800 has 2404352 pending on destroyed export [ 5348.833596] LustreError: 8435:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_statfs: tot_granted 239680 != fo_tot_granted 2644032 [ 5348.840575] LustreError: 8435:0:(tgt_grant.c:237:tgt_grant_sanity_check()) Skipped 2 previous similar messages [ 5348.844878] LustreError: 8435:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 2404352 [ 5348.853730] LustreError: 8435:0:(tgt_grant.c:240:tgt_grant_sanity_check()) Skipped 2 previous similar messages [ 5352.850499] LDISKFS-fs (dm-0): recovery complete [ 5352.856716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5353.142994] LustreError: 8434:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_statfs: tot_granted 239680 != fo_tot_granted 2644032 [ 5353.154319] LustreError: 8434:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 2404352 [ 5353.297547] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5353.300581] Lustre: lustre-MDT0000: Aborting client recovery [ 5353.306793] LustreError: 95110:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5353.308628] Lustre: Skipped 7 previous similar messages [ 5353.317322] Lustre: 95142:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5353.327644] Lustre: 95142:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c154a474-28d3-40b8-9d00-0def439c1563@ [ 5353.337479] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 5353.347869] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5353.367167] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5353.446438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1475 to 0x280000401:1505) [ 5353.452247] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1505) [ 5358.222903] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5358.577570] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5358.589600] LustreError: 8434:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_statfs: tot_granted 238784 != fo_tot_granted 2643136 [ 5358.601501] LustreError: 8434:0:(tgt_grant.c:237:tgt_grant_sanity_check()) Skipped 1 previous similar message [ 5358.616899] LustreError: 8434:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 2404352 [ 5358.626338] LustreError: 8434:0:(tgt_grant.c:240:tgt_grant_sanity_check()) Skipped 1 previous similar message [ 5363.134785] LustreError: 8435:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_statfs: tot_granted 8676544 != fo_tot_granted 11080896 [ 5363.139527] LustreError: 8435:0:(tgt_grant.c:237:tgt_grant_sanity_check()) Skipped 1 previous similar message [ 5363.146693] LustreError: 8435:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 2404352 [ 5363.151581] LustreError: 8435:0:(tgt_grant.c:240:tgt_grant_sanity_check()) Skipped 1 previous similar message [ 5373.944069] LustreError: 10903:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_statfs: tot_granted 8676544 != fo_tot_granted 11080896 [ 5373.953139] LustreError: 10903:0:(tgt_grant.c:237:tgt_grant_sanity_check()) Skipped 5 previous similar messages [ 5373.962703] LustreError: 10903:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 2404352 [ 5373.970209] LustreError: 10903:0:(tgt_grant.c:240:tgt_grant_sanity_check()) Skipped 5 previous similar messages [ 5386.753186] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 05:03:01 (1743498181) [ 5394.403906] LustreError: 8435:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_statfs: tot_granted 8676544 != fo_tot_granted 11080896 [ 5394.415466] LustreError: 8435:0:(tgt_grant.c:237:tgt_grant_sanity_check()) Skipped 7 previous similar messages [ 5394.423770] LustreError: 8435:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 2404352 [ 5394.435371] LustreError: 8435:0:(tgt_grant.c:240:tgt_grant_sanity_check()) Skipped 7 previous similar messages [ 5395.596747] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5397.561653] Lustre: Failing over lustre-MDT0000 [ 5397.808723] Lustre: server umount lustre-MDT0000 complete [ 5412.366664] LDISKFS-fs (dm-0): recovery complete [ 5412.368834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5412.483387] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5412.487992] LustreError: Skipped 7 previous similar messages [ 5412.741808] Lustre: *** cfs_fail_loc=1311, val=0*** [ 5412.776211] Lustre: lustre-MDT0000: Aborting client recovery [ 5412.778401] LustreError: 96954:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5412.783777] Lustre: 96986:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5412.791897] Lustre: 96986:0:(ldlm_lib.c:2289:target_recovery_overseer()) Skipped 2 previous similar messages [ 5412.797035] Lustre: 96986:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 5412.802558] Lustre: 96986:0:(genops.c:1508:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 5412.806891] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 5412.817214] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 5412.836587] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 5412.882040] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 5412.882602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 5417.232929] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5417.972846] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5428.202418] LustreError: 8434:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_statfs: tot_granted 8675648 != fo_tot_granted 11080000 [ 5428.221255] LustreError: 8434:0:(tgt_grant.c:237:tgt_grant_sanity_check()) Skipped 16 previous similar messages [ 5428.227895] LustreError: 8434:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 2404352 [ 5428.233292] LustreError: 8434:0:(tgt_grant.c:240:tgt_grant_sanity_check()) Skipped 16 previous similar messages [ 5433.279539] Lustre: *** cfs_fail_loc=1311, val=0*** [ 5433.282410] Lustre: Skipped 1 previous similar message [ 5443.911345] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 05:03:59 (1743498239) [ 5453.174904] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5455.414890] Lustre: Failing over lustre-MDT0000 [ 5455.637120] Lustre: server umount lustre-MDT0000 complete [ 5470.669536] LDISKFS-fs (dm-0): recovery complete [ 5470.671946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5471.129867] Lustre: lustre-MDT0000: Aborting client recovery [ 5471.132021] LustreError: 98801:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5471.138401] Lustre: 98833:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5471.144558] Lustre: 98833:0:(ldlm_lib.c:2289:target_recovery_overseer()) Skipped 2 previous similar messages [ 5471.150153] Lustre: 98833:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 5471.157903] Lustre: 98833:0:(genops.c:1508:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 5471.161540] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 5471.176042] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 5471.193042] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 5471.241988] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 5471.242444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 5475.680235] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5476.342080] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5496.807045] LustreError: 10903:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_statfs: tot_granted 8674752 != fo_tot_granted 11079104 [ 5496.813144] LustreError: 10903:0:(tgt_grant.c:237:tgt_grant_sanity_check()) Skipped 28 previous similar messages [ 5496.816526] LustreError: 10903:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 2404352 [ 5496.819796] LustreError: 10903:0:(tgt_grant.c:240:tgt_grant_sanity_check()) Skipped 28 previous similar messages [ 5503.398863] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 05:04:58 (1743498298) [ 5504.590743] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5504.597911] LustreError: 6556:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e9517491d00 x1828185025485440/t201863462916(0) o36->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:218/0 lens 512/456 e 0 to 0 dl 1743498313 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 [ 5508.778232] Lustre: Failing over lustre-MDT0000 [ 5508.970419] Lustre: server umount lustre-MDT0000 complete [ 5519.276550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5519.639692] Lustre: lustre-MDT0000: Aborting client recovery [ 5519.641660] LustreError: 100105:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5519.652307] Lustre: 100138:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5519.658609] Lustre: 100138:0:(ldlm_lib.c:2289:target_recovery_overseer()) Skipped 2 previous similar messages [ 5519.666392] Lustre: 100138:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 5519.672736] Lustre: 100138:0:(genops.c:1508:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 5519.678290] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 5519.685518] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 5519.707431] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 5519.748157] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 5519.752901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1601) [ 5524.169181] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5524.970780] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5550.981837] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 5553.899727] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 05:05:49 (1743498349) [ 5562.969535] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5566.098212] Lustre: Failing over lustre-MDT0000 [ 5566.392187] Lustre: server umount lustre-MDT0000 complete [ 5581.004593] LDISKFS-fs (dm-0): recovery complete [ 5581.007347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.283304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 5581.535411] Lustre: lustre-MDT0000: Aborting client recovery [ 5581.538711] LustreError: 102049:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5581.547080] Lustre: 102081:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5581.553431] Lustre: 102081:0:(ldlm_lib.c:2289:target_recovery_overseer()) Skipped 2 previous similar messages [ 5581.561572] Lustre: 102081:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 5581.567144] Lustre: 102081:0:(genops.c:1508:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 5581.571650] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 5581.582783] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 5581.597518] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 5581.644842] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 5581.654747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1633) [ 5585.996463] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5586.929853] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 5613.868620] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 05:06:49 (1743498409) [ 5625.314118] LustreError: 8435:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_statfs: tot_granted 229824 != fo_tot_granted 2634176 [ 5625.322613] LustreError: 8435:0:(tgt_grant.c:237:tgt_grant_sanity_check()) Skipped 63 previous similar messages [ 5625.327622] LustreError: 8435:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 2404352 [ 5625.342937] LustreError: 8435:0:(tgt_grant.c:240:tgt_grant_sanity_check()) Skipped 63 previous similar messages [ 5656.312464] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5658.724267] Lustre: Failing over lustre-MDT0000 [ 5659.029057] Lustre: server umount lustre-MDT0000 complete [ 5680.096286] Lustre: 3682:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743498461/real 1743498461] req@ffff9e95163dcb00 x1828185065509760/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743498477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 5680.123424] Lustre: 3682:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 5682.959243] LDISKFS-fs (dm-0): recovery complete [ 5682.961693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5690.336521] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e963f7f8600 x1828185065518464/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 5690.352118] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 5690.690899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5690.701167] Lustre: Skipped 8 previous similar messages [ 5691.894751] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5691.897412] Lustre: Skipped 7 previous similar messages [ 5695.040794] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5696.070304] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5696.079206] Lustre: Skipped 7 previous similar messages [ 5696.104916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 5696.106646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 5704.890368] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5707.667851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5738.203984] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 05:08:53 (1743498533) [ 5771.648943] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5785.058032] Lustre: Failing over lustre-MDT0000 [ 5785.157985] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5785.169837] LustreError: Skipped 5 previous similar messages [ 5785.541524] Lustre: server umount lustre-MDT0000 complete [ 5788.129172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5788.138200] Lustre: Skipped 31 previous similar messages [ 5811.907872] LDISKFS-fs (dm-0): recovery complete [ 5811.910777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5814.112144] LustreError: 106532:0:(import.c:333:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 5814.115243] LustreError: 106532:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9e95163d8600 x1828185065681280/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1743498612 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 5814.135027] LustreError: 106532:0:(import.c:367:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 5814.767567] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb7fdae [ 5819.407915] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5824.367120] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 5824.373807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 5829.467775] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5832.376671] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5861.278484] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 05:10:56 (1743498656) [ 5864.043620] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5865.106655] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 5865.111795] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5875.603974] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 05:11:10 (1743498670) [ 5885.924218] LustreError: 8434:0:(tgt_grant.c:237:tgt_grant_sanity_check()) ofd_statfs: tot_granted 17055168 != fo_tot_granted 19459520 [ 5885.929745] LustreError: 8434:0:(tgt_grant.c:237:tgt_grant_sanity_check()) Skipped 107 previous similar messages [ 5885.933362] LustreError: 8434:0:(tgt_grant.c:240:tgt_grant_sanity_check()) ofd_statfs: tot_pending 0 != fo_tot_pending 2404352 [ 5885.945262] LustreError: 8434:0:(tgt_grant.c:240:tgt_grant_sanity_check()) Skipped 107 previous similar messages [ 5908.984159] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5923.232555] Lustre: Failing over lustre-OST0000 [ 5923.320399] Lustre: server umount lustre-OST0000 complete [ 5926.888467] LustreError: 39089:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5926.895867] LustreError: 39089:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 136 previous similar messages [ 5946.967702] LDISKFS-fs (dm-2): recovery complete [ 5946.971143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5947.064828] Lustre: 109203:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 5950.070704] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 5950.078388] Lustre: Skipped 34 previous similar messages [ 5953.697390] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6018.868827] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 05:13:34 (1743498814) [ 6027.761750] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6030.712143] Lustre: Failing over lustre-MDT0000 [ 6031.003715] Lustre: server umount lustre-MDT0000 complete [ 6050.785467] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6050.790781] LustreError: Skipped 5 previous similar messages [ 6054.493220] LDISKFS-fs (dm-0): recovery complete [ 6054.495562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6060.004969] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb9d619 [ 6060.330714] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6060.342202] Lustre: Skipped 17 previous similar messages [ 6064.306620] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6065.725950] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 6065.726325] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 6073.489275] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6076.260836] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6080.994713] LustreError: 111240:0:(osp_precreate.c:972:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 6080.997268] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6082.016563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 6099.442433] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 05:14:54 (1743498894) [ 6105.509878] LustreError: 18156:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_race id 701 sleeping [ 6110.688135] LustreError: 18156:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 6110.692873] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnecting [ 6110.763585] LustreError: 42084:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_fail_race id 701 waking [ 6112.603100] LustreError: 6556:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_race id 701 sleeping [ 6117.857711] LustreError: 6556:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 6117.862913] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnecting [ 6119.859060] LustreError: 6556:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_race id 701 sleeping [ 6125.024132] LustreError: 6556:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 6125.029466] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnecting [ 6126.826323] LustreError: 18156:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_race id 701 sleeping [ 6132.196841] LustreError: 18156:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 6134.055775] LustreError: 8910:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_race id 701 sleeping [ 6139.361814] LustreError: 8910:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 6139.365568] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnecting [ 6139.375890] Lustre: Skipped 1 previous similar message [ 6148.726318] LustreError: 18156:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_race id 701 sleeping [ 6148.731926] LustreError: 18156:0:(ldlm_lib.c:1081:target_handle_connect()) Skipped 1 previous similar message [ 6154.208949] LustreError: 18156:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 6154.220781] LustreError: 18156:0:(ldlm_lib.c:1081:target_handle_connect()) Skipped 1 previous similar message [ 6161.376310] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnecting [ 6161.384286] Lustre: Skipped 2 previous similar messages [ 6170.524596] LustreError: 6557:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_race id 701 sleeping [ 6170.529050] LustreError: 6557:0:(ldlm_lib.c:1081:target_handle_connect()) Skipped 2 previous similar messages [ 6175.712158] LustreError: 6557:0:(ldlm_lib.c:1081:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 6175.717766] LustreError: 6557:0:(ldlm_lib.c:1081:target_handle_connect()) Skipped 2 previous similar messages [ 6188.962755] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 05:16:24 (1743498984) [ 6191.386618] LustreError: 6558:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 6200.603046] Lustre: lustre-MDT0000: Export ffff9e95076a0800 already connecting from 192.168.206.51@tcp [ 6204.632076] Lustre: lustre-MDT0000: Export ffff9e95076a0800 already connecting from 192.168.206.51@tcp [ 6209.814307] Lustre: lustre-MDT0000: Export ffff9e95076a0800 already connecting from 192.168.206.51@tcp [ 6214.211679] Lustre: lustre-MDT0000: Export ffff9e95076a0800 already connecting from 192.168.206.51@tcp [ 6219.535526] Lustre: lustre-MDT0000: Export ffff9e95076a0800 already connecting from 192.168.206.51@tcp [ 6229.775281] Lustre: lustre-MDT0000: Export ffff9e95076a0800 already connecting from 192.168.206.51@tcp [ 6229.788028] Lustre: Skipped 1 previous similar message [ 6231.440091] LustreError: 6558:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 awake [ 6231.449273] Lustre: 6558:0:(service.c:2350:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9e95163df900 x1828185028154496/t0(0) o38->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:0/0 lens 520/416 e 0 to 0 dl 1743499009 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 [ 6234.899042] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnecting [ 6234.905868] Lustre: Skipped 3 previous similar messages [ 6234.907718] LustreError: 6558:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 6259.477705] Lustre: lustre-MDT0000: Export ffff9e95076a0800 already connecting from 192.168.206.51@tcp [ 6275.000558] LustreError: 6558:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 awake [ 6275.005994] Lustre: 6558:0:(service.c:2350:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff9e9514c4e200 x1828185028158976/t0(0) o38->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:0/0 lens 520/416 e 0 to 0 dl 1743499052 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 [ 6279.950878] LustreError: 8910:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 6304.529803] Lustre: lustre-MDT0000: Export ffff9e95076a0800 already connecting from 192.168.206.51@tcp [ 6304.540266] Lustre: Skipped 4 previous similar messages [ 6320.040141] LustreError: 8910:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 awake [ 6320.045817] Lustre: 8910:0:(service.c:2350:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff9e9618275c40 x1828185028162176/t0(0) o38->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:0/0 lens 520/416 e 0 to 0 dl 1743499097 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 [ 6325.010780] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnecting [ 6325.018602] Lustre: Skipped 1 previous similar message [ 6325.021583] LustreError: 15213:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 6365.112131] LustreError: 15213:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 awake [ 6365.120727] Lustre: 15213:0:(service.c:2350:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9e9505899180 x1828185028165760/t0(0) o38->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:0/0 lens 520/416 e 0 to 0 dl 1743499143 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 [ 6370.065715] LustreError: 15213:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 6394.644096] Lustre: lustre-MDT0000: Export ffff9e95076a0800 already connecting from 192.168.206.51@tcp [ 6394.652816] Lustre: Skipped 9 previous similar messages [ 6410.129455] LustreError: 15213:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 awake [ 6410.133555] Lustre: 15213:0:(service.c:2350:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9e9514dc3f80 x1828185028169344/t0(0) o38->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:0/0 lens 520/416 e 0 to 0 dl 1743499188 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 [ 6414.612309] LustreError: 8910:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 6422.081463] LustreError: 8910:0:(ldlm_lib.c:1334:target_handle_connect()) cfs_fail_timeout interrupted [ 6429.503441] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 05:20:24 (1743499224) [ 6437.363334] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6441.913891] Lustre: Failing over lustre-MDT0000 [ 6442.128365] Lustre: server umount lustre-MDT0000 complete [ 6444.515456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6444.525027] Lustre: Skipped 12 previous similar messages [ 6444.527938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6444.532303] LustreError: Skipped 4 previous similar messages [ 6454.586079] LDISKFS-fs (dm-0): recovery complete [ 6454.589203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.756094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 6454.765556] Lustre: Skipped 6 previous similar messages [ 6454.857067] Lustre: *** cfs_fail_loc=712, val=0*** [ 6454.864189] LustreError: 42086:0:(service.c:1219:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9e9517497900 x1828185066202624/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 [ 6454.937103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.939844] Lustre: Skipped 3 previous similar messages [ 6454.991402] Lustre: lustre-MDT0000: Aborting client recovery [ 6454.993341] LustreError: 115980:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 6454.995065] Lustre: 116012:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6455.004194] Lustre: 116012:0:(ldlm_lib.c:2289:target_recovery_overseer()) Skipped 2 previous similar messages [ 6455.007789] Lustre: 116012:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 6455.012360] Lustre: 116012:0:(genops.c:1508:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6455.015770] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 6455.025868] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 6455.038742] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 6455.086247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 6455.112891] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 6458.549547] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6460.390746] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 6474.756342] Lustre: Failing over lustre-MDT0000 [ 6474.954545] Lustre: server umount lustre-MDT0000 complete [ 6491.104157] Lustre: 3680:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743499273/real 1743499273] req@ffff9e95134e5c40 x1828185066220032/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743499289 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 6491.115166] Lustre: 3680:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6493.352268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6501.360248] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacb9e9c9 [ 6501.650113] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6501.660688] Lustre: Skipped 3 previous similar messages [ 6505.539287] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6507.008099] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 6507.018025] Lustre: Skipped 3 previous similar messages [ 6507.050501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 6507.051864] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 6514.155243] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6516.756377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6528.625800] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 05:22:04 (1743499324) [ 6529.016624] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnecting [ 6529.021270] Lustre: Skipped 2 previous similar messages [ 6539.466945] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 05:22:14 (1743499334) [ 6540.607565] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 6540.609853] LustreError: 6565:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e963a136200 x1828185028234752/t0(0) o700->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:499/0 lens 264/248 e 0 to 0 dl 1743499349 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 [ 6560.608917] Lustre: Failing over lustre-MDT0000 [ 6560.826252] Lustre: server umount lustre-MDT0000 complete [ 6562.069772] LustreError: 6557:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6562.079059] LustreError: 6557:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 86 previous similar messages [ 6578.945252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6582.791034] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6584.806695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 6584.813055] Lustre: Skipped 16 previous similar messages [ 6584.850067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 6584.850139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 6590.817478] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6593.206201] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6606.090367] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 05:23:21 (1743499401) [ 6608.975456] Lustre: Failing over lustre-OST0000 [ 6609.106802] Lustre: server umount lustre-OST0000 complete [ 6626.609833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6626.675340] Lustre: 119784:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 6631.916397] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6639.912923] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6642.469486] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6717.140418] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 05:25:12 (1743499512) [ 6725.122828] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6727.341154] Lustre: Failing over lustre-MDT0000 [ 6727.581143] Lustre: server umount lustre-MDT0000 complete [ 6745.569242] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6745.574235] LustreError: Skipped 3 previous similar messages [ 6749.256684] LDISKFS-fs (dm-0): recovery complete [ 6749.259801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.809024] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e963a131740 x1828185066364928/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 6755.826856] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) Skipped 3 previous similar messages [ 6756.074397] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6756.076969] Lustre: Skipped 6 previous similar messages [ 6759.356894] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6761.827996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 6761.828047] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 6761.867058] Lustre: *** cfs_fail_loc=216, val=0*** [ 6761.869068] LustreError: 121772:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -30 [ 6761.873365] LustreError: 121772:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -30 [ 6835.312856] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 05:27:10 (1743499630) [ 6837.561737] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 6837.564821] Lustre: Skipped 2 previous similar messages [ 6852.878111] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 05:27:28 (1743499648) [ 6856.338276] Lustre: Failing over lustre-MDT0000 [ 6856.564270] Lustre: server umount lustre-MDT0000 complete [ 6875.288759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6888.059973] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6889.974890] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 6889.977041] LustreError: 123338:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e950589ae40 x1828185028366848/t0(0) o101->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:132/0 lens 328/344 e 0 to 0 dl 1743499737 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 [ 6945.038696] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 6945.081421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 6945.081670] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 6950.401497] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6952.745872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6964.974296] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 05:29:20 (1743499760) [ 6967.412161] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 6975.254668] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6977.088948] Lustre: Failing over lustre-MDT0000 [ 6977.310165] Lustre: server umount lustre-MDT0000 complete [ 6999.407530] LDISKFS-fs (dm-0): recovery complete [ 6999.409908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7008.743486] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacba1b32 [ 7012.613264] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7014.404437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3137) [ 7014.404955] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3105) [ 7020.334333] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7022.967611] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7034.615539] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 05:30:30 (1743499830) [ 7036.133143] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 7045.121739] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7046.994253] Lustre: Failing over lustre-MDT0000 [ 7047.150903] Lustre: server umount lustre-MDT0000 complete [ 7050.211153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7050.212355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7050.221692] LustreError: Skipped 5 previous similar messages [ 7050.235633] Lustre: Skipped 27 previous similar messages [ 7069.383624] LDISKFS-fs (dm-0): recovery complete [ 7069.385573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7077.047151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7077.051471] Lustre: Skipped 6 previous similar messages [ 7080.740828] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7082.526240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 7082.527725] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3137) [ 7089.418365] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7092.147555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7104.077171] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 05:31:39 (1743499899) [ 7105.480744] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 7114.275472] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7116.102388] Lustre: Failing over lustre-MDT0000 [ 7116.293223] Lustre: server umount lustre-MDT0000 complete [ 7134.691933] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743499916/real 1743499916] req@ffff9e9517c34b00 x1828185066562432/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743499932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 7134.716148] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 7138.704943] LDISKFS-fs (dm-0): recovery complete [ 7138.708401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7145.642839] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7145.646351] Lustre: Skipped 6 previous similar messages [ 7148.988780] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7150.596605] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7150.606257] Lustre: Skipped 6 previous similar messages [ 7150.644673] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 7150.644689] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3201) [ 7164.414110] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 05:32:39 (1743499959) [ 7167.054619] Lustre: *** cfs_fail_loc=13b, val=315*** [ 7167.059058] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 7167.076905] LustreError: 6560:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e9507dee7c0 x1828185028432512/t257698037777(0) o35->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:371/0 lens 392/456 e 0 to 0 dl 1743499976 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 7169.801902] Lustre: Failing over lustre-MDT0000 [ 7170.057327] Lustre: server umount lustre-MDT0000 complete [ 7170.850572] LustreError: 15213:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7170.858407] LustreError: 15213:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 200 previous similar messages [ 7188.179749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.242515] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7203.308781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 7203.313420] Lustre: Skipped 29 previous similar messages [ 7203.350343] Lustre: 6560:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e95177ef340 x1828185028432512/t257698037777(0) o35->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:407/0 lens 392/456 e 0 to 0 dl 1743500012 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 7203.351309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3233) [ 7203.355943] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 7209.194222] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7211.838049] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7223.628132] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 05:33:39 (1743500019) [ 7225.206666] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7225.208788] LustreError: 15213:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e95177567c0 x1828185028448640/t261993005072(0) o36->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:473/0 lens 504/448 e 0 to 0 dl 1743500078 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 7234.505936] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7236.305710] Lustre: Failing over lustre-MDT0000 [ 7236.474758] Lustre: server umount lustre-MDT0000 complete [ 7257.929670] LDISKFS-fs (dm-0): recovery complete [ 7257.931686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7265.766942] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacba3461 [ 7265.771097] Lustre: Skipped 1 previous similar message [ 7269.567471] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7271.438803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 7271.439908] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233) [ 7271.459659] Lustre: 8910:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e963a0e50c0 x1828185028448640/t261993005072(0) o36->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:519/0 lens 504/2880 e 0 to 0 dl 1743500124 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 7277.242564] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7279.803694] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7291.868967] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 05:34:47 (1743500087) [ 7293.257934] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7293.262161] LustreError: 15213:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e950520f900 x1828185028465152/t266287972368(0) o36->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:541/0 lens 504/448 e 0 to 0 dl 1743500146 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 7295.127484] Lustre: *** cfs_fail_loc=13b, val=315*** [ 7302.279305] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7304.309793] Lustre: Failing over lustre-MDT0000 [ 7304.468995] Lustre: server umount lustre-MDT0000 complete [ 7326.477816] LDISKFS-fs (dm-0): recovery complete [ 7326.480615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7337.317143] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7339.526863] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 7339.527198] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3297) [ 7339.531868] Lustre: 6557:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e9517c35c40 x1828185028465152/t266287972368(0) o36->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:587/0 lens 504/2880 e 0 to 0 dl 1743500192 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 7351.874662] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 05:35:47 (1743500147) [ 7353.195963] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7353.198978] Lustre: Skipped 1 previous similar message [ 7353.200895] LustreError: 6558:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e9514c48bc0 x1828185028480000/t270582939664(0) o36->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:601/0 lens 504/448 e 0 to 0 dl 1743500206 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 7353.211496] LustreError: 6558:0:(ldlm_lib.c:3251:target_send_reply_msg()) Skipped 1 previous similar message [ 7355.039780] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 7355.042199] Lustre: Skipped 1 previous similar message [ 7362.644600] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7364.592554] Lustre: Failing over lustre-MDT0000 [ 7364.744224] Lustre: server umount lustre-MDT0000 complete [ 7381.472502] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7381.476683] LustreError: Skipped 7 previous similar messages [ 7386.474385] LDISKFS-fs (dm-0): recovery complete [ 7386.476963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7392.037793] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7392.043816] Lustre: Skipped 7 previous similar messages [ 7395.635427] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7397.398145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3329) [ 7397.398170] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 7397.419630] Lustre: 8910:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e95134e5c40 x1828185028480000/t270582939664(0) o36->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:645/0 lens 504/2880 e 0 to 0 dl 1743500250 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 7397.434507] Lustre: 8910:0:(mdt_recovery.c:128:mdt_req_from_lrd()) Skipped 1 previous similar message [ 7409.300396] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 05:36:44 (1743500204) [ 7410.691516] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 7412.437759] Lustre: *** cfs_fail_loc=13b, val=315*** [ 7412.439688] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 7412.442396] LustreError: 6559:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e963f7e4540 x1828185028494592/t274877906960(0) o35->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:616/0 lens 392/456 e 0 to 0 dl 1743500221 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 7419.954703] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7421.918368] Lustre: Failing over lustre-MDT0000 [ 7422.106757] Lustre: server umount lustre-MDT0000 complete [ 7443.821553] LDISKFS-fs (dm-0): recovery complete [ 7443.823631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.467235] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7454.217865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 7454.222659] Lustre: 6559:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e962fd7e200 x1828185028494592/t274877906960(0) o35->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:658/0 lens 392/456 e 0 to 0 dl 1743500263 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 7454.223185] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 7466.192659] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 05:37:41 (1743500261) [ 7467.336853] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 7467.340812] LustreError: 6557:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e951766cb00 x1828185028507136/t279172874255(0) o101->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:715/0 lens 664/608 e 0 to 0 dl 1743500320 ref 1 fl Interpret:/200/0 rc 301/0 job:'touch.0' uid:0 gid:0 [ 7526.673784] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnecting [ 7526.679867] Lustre: Skipped 1 previous similar message [ 7526.692968] Lustre: 6557:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e951616cb00 x1828185028507136/t279172874255(0) o101->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:19/0 lens 664/3488 e 0 to 0 dl 1743500379 ref 1 fl Interpret:/202/0 rc 0/0 job:'touch.0' uid:0 gid:0 [ 7536.224177] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 05:38:51 (1743500331) [ 7543.549691] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7545.568200] Lustre: Failing over lustre-MDT0000 [ 7545.741287] Lustre: server umount lustre-MDT0000 complete [ 7567.773047] LDISKFS-fs (dm-0): recovery complete [ 7567.776650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.932905] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7578.676704] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 7578.677536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3393) [ 7585.569456] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7588.147467] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7609.422641] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 05:40:05 (1743500405) [ 7617.297303] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7619.068054] Lustre: Failing over lustre-MDT0000 [ 7619.266310] Lustre: server umount lustre-MDT0000 complete [ 7639.922039] LDISKFS-fs (dm-0): recovery complete [ 7639.925079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7650.272968] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e963f6150c0 x1828185066839808/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 7650.281796] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) Skipped 14 previous similar messages [ 7653.836289] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7655.978428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 7655.979532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3393) [ 7661.408811] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7663.895756] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7668.723720] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 7682.545989] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 05:41:18 (1743500478) [ 7726.152762] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7727.637502] Lustre: Failing over lustre-MDT0000 [ 7728.192764] Lustre: server umount lustre-MDT0000 complete [ 7732.709565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7732.713101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7732.728515] Lustre: Skipped 34 previous similar messages [ 7732.732112] LustreError: Skipped 5 previous similar messages [ 7748.065213] Lustre: 3682:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743500530/real 1743500530] req@ffff9e95055dbf80 x1828185066895360/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743500546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 7748.075123] Lustre: 3682:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 7748.952184] LDISKFS-fs (dm-0): recovery complete [ 7748.954858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.464641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7758.469170] Lustre: Skipped 8 previous similar messages [ 7759.119504] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7759.122215] Lustre: Skipped 7 previous similar messages [ 7761.732650] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7764.187521] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 7764.191022] Lustre: Skipped 7 previous similar messages [ 7764.219818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 7764.224233] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 7769.091898] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7771.300249] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7824.444861] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 05:43:39 (1743500619) [ 7833.592843] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7835.601149] Lustre: Failing over lustre-MDT0000 [ 7835.630218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7835.633889] Lustre: Skipped 3 previous similar messages [ 7835.847179] Lustre: server umount lustre-MDT0000 complete [ 7836.448456] LustreError: 6557:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7836.456456] LustreError: 6557:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 266 previous similar messages [ 7857.152151] LDISKFS-fs (dm-0): recovery complete [ 7857.154755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7869.745759] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7871.982530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 7871.985910] Lustre: Skipped 34 previous similar messages [ 7872.037858] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 7872.038241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 7877.230869] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7879.540976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7887.676978] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 7889.945736] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 7898.629201] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 05:44:54 (1743500694) [ 7906.573166] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 7969.475209] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 7969.480257] Lustre: Skipped 1 previous similar message [ 7969.482415] LustreError: 15213:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e962f18a050 x1828185031316608/t296352743435(0) o36->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:462/0 lens 66040/440 e 0 to 0 dl 1743500822 ref 1 fl Interpret:/200/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 [ 8030.501536] Lustre: 6556:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e9517491740 x1828185031316608/t296352743435(0) o36->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:523/0 lens 66040/440 e 0 to 0 dl 1743500883 ref 1 fl Interpret:/202/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 [ 8041.949277] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 8044.217214] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 05:47:19 (1743500839) [ 8057.847889] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8060.398951] Lustre: Failing over lustre-MDT0000 [ 8060.680165] Lustre: server umount lustre-MDT0000 complete [ 8077.792536] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8077.797827] LustreError: Skipped 5 previous similar messages [ 8081.790202] LDISKFS-fs (dm-0): recovery complete [ 8081.792453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8087.278093] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8087.283152] Lustre: Skipped 5 previous similar messages [ 8090.720646] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8093.334804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 8093.334806] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 8098.431875] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8100.779183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8113.357358] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 05:48:29 (1743500909) [ 8135.890174] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 8147.759611] Lustre: Failing over lustre-OST0000 [ 8147.844923] Lustre: server umount lustre-OST0000 complete [ 8169.416497] LDISKFS-fs (dm-2): recovery complete [ 8169.421081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8169.504434] Lustre: 151083:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8174.980112] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8188.987798] Lustre: Failing over lustre-OST0000 [ 8189.075490] Lustre: server umount lustre-OST0000 complete [ 8207.015411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8207.091887] Lustre: 152064:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8212.460724] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8220.628328] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8223.093780] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8266.353438] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 05:51:02 (1743501062) [ 8268.869341] Lustre: Failing over lustre-MDT0000 [ 8269.131493] Lustre: server umount lustre-MDT0000 complete [ 8287.124269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.931676] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8292.873484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 8292.873495] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 8304.941788] Lustre: Failing over lustre-MDT0000 [ 8305.144750] Lustre: server umount lustre-MDT0000 complete [ 8323.276411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8326.766711] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8328.709482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 8328.710774] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 8334.276670] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8336.644580] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8347.364270] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 05:52:23 (1743501143) [ 8361.866175] Lustre: Failing over lustre-OST0000 [ 8361.936965] Lustre: server umount lustre-OST0000 complete [ 8362.979419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8362.982490] LustreError: Skipped 8 previous similar messages [ 8362.985753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8362.991803] Lustre: Skipped 20 previous similar messages [ 8378.625881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8378.687330] Lustre: 155689:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 8378.806501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8378.813898] Lustre: Skipped 6 previous similar messages [ 8380.003736] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8380.007899] Lustre: Skipped 6 previous similar messages [ 8380.104873] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8380.110615] Lustre: Skipped 6 previous similar messages [ 8383.158447] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8390.065905] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8392.132808] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8403.386811] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 05:53:19 (1743501199) [ 8405.455209] Lustre: Failing over lustre-MDT0000 [ 8405.491849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8405.788227] Lustre: server umount lustre-MDT0000 complete [ 8413.368099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8413.452768] Lustre: *** cfs_fail_loc=605, val=0*** [ 8413.454715] LustreError: 157113:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc087d400 failed: rc = -95 [ 8413.460176] LustreError: 157113:0:(obd_config.c:780:class_setup()) setup MGS failed (-95) [ 8413.462682] LustreError: 157113:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 8413.466132] LustreError: 157113:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 8413.469650] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 8413.472324] LustreError: 157113:0:(tgt_mount.c:1761:server_put_super()) no obd lustre-MDT0000 [ 8413.479901] Lustre: server umount lustre-MDT0000 complete [ 8413.481858] LustreError: 157113:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 8419.845295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8423.113159] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8425.487237] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 8425.487588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 8431.950267] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 05:53:47 (1743501227) [ 8437.458605] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 8439.670269] Lustre: Failing over lustre-MDT0000 [ 8439.946995] Lustre: server umount lustre-MDT0000 complete [ 8440.804498] LustreError: 7825:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8440.810449] LustreError: 7825:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 147 previous similar messages [ 8457.184178] Lustre: 3681:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743501238/real 1743501238] req@ffff9e96292c22c0 x1828185068051200/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743501254 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 8457.196157] Lustre: 3681:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 8458.886507] LDISKFS-fs (dm-0): recovery complete [ 8458.889367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8467.428033] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacbfae52 [ 8467.432175] Lustre: Skipped 2 previous similar messages [ 8467.731131] Lustre: *** cfs_fail_loc=707, val=0*** [ 8470.364080] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 8473.066641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 8473.069598] Lustre: Skipped 26 previous similar messages [ 8528.161759] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 8528.498665] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4942 to 0x2c0000401:4961) [ 8528.498968] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4975 to 0x280000401:4993) [ 8532.617448] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8534.668590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8545.672982] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 05:55:41 (1743501341) [ 8574.294035] LustreError: 6558:0:(service.c:2329:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 8585.320154] LustreError: 6558:0:(service.c:2329:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 8605.345656] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 05:56:41 (1743501401) [ 8634.776115] LustreError: 41269:0:(tgt_handler.c:2745:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 8645.816446] LustreError: 41269:0:(tgt_handler.c:2745:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 8656.770694] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 05:57:32 (1743501452) [ 8684.288934] LustreError: 8910:0:(service.c:2329:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 8689.296268] LustreError: 8910:0:(service.c:2329:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 8701.464133] LustreError: 8910:0:(service.c:2329:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 8720.670484] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 05:58:36 (1743501516) [ 8814.578228] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 06:00:10 (1743501610) [ 8841.289550] LustreError: 18156:0:(service.c:2329:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 8841.292603] LustreError: 18156:0:(service.c:2329:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 8841.712147] LustreError: 18156:0:(service.c:2329:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 8857.568688] LustreError: 10903:0:(service.c:2329:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 8857.571660] LustreError: 10903:0:(service.c:2329:ptlrpc_server_handle_request()) Skipped 55 previous similar messages [ 8857.984142] LustreError: 8435:0:(service.c:2329:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 8857.986897] LustreError: 8435:0:(service.c:2329:ptlrpc_server_handle_request()) Skipped 54 previous similar messages [ 8891.655674] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 06:01:27 (1743501687) [ 8921.522481] Lustre: DEBUG MARKER: phase 2 [ 8931.666609] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 06:02:07 (1743501727) [ 9013.648670] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 06:03:29 (1743501809) [ 9015.478347] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 9017.832635] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 06:03:33 (1743501813) [ 9022.519400] Lustre: DEBUG MARKER: Started rundbench load pid=132958 ... [ 9030.313411] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9033.245984] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 9035.151522] Lustre: Failing over lustre-MDT0000 [ 9035.410729] Lustre: server umount lustre-MDT0000 complete [ 9036.257510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9036.262099] Lustre: Skipped 11 previous similar messages [ 9041.167206] LustreError: 15213:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9041.176433] LustreError: 15213:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 34 previous similar messages [ 9052.640138] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9052.644966] LustreError: Skipped 4 previous similar messages [ 9054.993214] LDISKFS-fs (dm-0): recovery complete [ 9054.995797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9062.884839] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacc1fcba [ 9063.078832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9063.082795] Lustre: Skipped 2 previous similar messages [ 9063.115343] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9063.117334] Lustre: Skipped 7 previous similar messages [ 9064.761463] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9064.763241] Lustre: Skipped 2 previous similar messages [ 9066.067633] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9068.559772] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9068.563310] Lustre: Skipped 2 previous similar messages [ 9068.578442] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 9068.578532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 9073.805115] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9076.296247] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9085.390605] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9088.503867] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 9090.431260] Lustre: Failing over lustre-MDT0001 [ 9090.694148] Lustre: server umount lustre-MDT0001 complete [ 9094.113361] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 9094.116507] LustreError: Skipped 3 previous similar messages [ 9108.757614] LDISKFS-fs (dm-1): recovery complete [ 9108.759569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9108.783879] Lustre: 168519:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9111.275685] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9114.088537] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 9114.091268] Lustre: Skipped 8 previous similar messages [ 9114.446316] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:725 to 0x280000400:769) [ 9114.446631] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:708 to 0x2c0000400:737) [ 9118.562772] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9121.171149] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9130.627539] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9133.858662] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 9135.801356] Lustre: Failing over lustre-MDT0000 [ 9135.971190] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 9136.074587] Lustre: server umount lustre-MDT0000 complete [ 9155.697753] LDISKFS-fs (dm-0): recovery complete [ 9155.700082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9158.923234] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9161.233260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 9161.233261] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 9166.333833] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9168.989522] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9178.319038] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9181.436890] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 9183.230914] Lustre: Failing over lustre-MDT0001 [ 9183.238912] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 9183.468657] Lustre: server umount lustre-MDT0001 complete [ 9201.494322] LDISKFS-fs (dm-1): recovery complete [ 9201.495832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9201.548433] Lustre: 171847:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9204.108644] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9207.780346] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1065 to 0x280000400:1089) [ 9207.780777] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1034 to 0x2c0000400:1057) [ 9212.783769] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9216.062910] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9226.114511] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9229.581077] Lustre: DEBUG MARKER: test_70b fail mds1 5 times [ 9231.912924] Lustre: Failing over lustre-MDT0000 [ 9231.992140] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 9232.128989] Lustre: server umount lustre-MDT0000 complete [ 9248.672087] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743502030/real 1743502030] req@ffff9e963a230bc0 x1828185069823616/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743502046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 9248.680856] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9252.625864] LDISKFS-fs (dm-0): recovery complete [ 9252.629653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9258.979579] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9515fda2c0 x1828185070078208/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [ 9258.988314] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [ 9262.235857] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9264.674173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5153) [ 9264.674218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5121) [ 9270.114118] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9272.816718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9282.097651] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9285.166247] Lustre: DEBUG MARKER: test_70b fail mds2 6 times [ 9287.224695] Lustre: Failing over lustre-MDT0001 [ 9287.405802] Lustre: server umount lustre-MDT0001 complete [ 9305.340949] LDISKFS-fs (dm-1): recovery complete [ 9305.342849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9305.398214] Lustre: 175173:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9308.132437] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9311.643125] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1396 to 0x280000400:1441) [ 9311.643319] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1364 to 0x2c0000400:1409) [ 9315.854204] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9318.481794] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9389.522201] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 06:09:45 (1743502185) [ 9516.828941] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9529.435825] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 9530.968802] Lustre: Failing over lustre-MDT0000 [ 9530.983329] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 9531.231604] Lustre: server umount lustre-MDT0000 complete [ 9531.887832] LustreError: 144817:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743502329 with bad export cookie 7018337741247738802 [ 9531.892865] LustreError: 144817:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9549.403378] LDISKFS-fs (dm-0): recovery complete [ 9549.405610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9552.232897] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9557.870718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5782 to 0x280000401:5825) [ 9557.871714] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5750 to 0x2c0000401:5793) [ 9562.264430] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9564.961632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9693.674774] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 9706.475910] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 9707.942518] Lustre: Failing over lustre-MDT0000 [ 9707.945618] LustreError: lustre-MDT0000-osp-MDT0001: operation ldlm_cancel to node 0@lo failed: rc = -19 [ 9707.948656] LustreError: Skipped 3 previous similar messages [ 9707.948826] LustreError: 178232:0:(ldlm_resource.c:983:ldlm_resource_complain()) lustre-MDT0001-osp-MDT0000: namespace resource [0x240001f58:0x1b:0x0].0x0 (ffff9e9507fd1080) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9707.949963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9707.958655] Lustre: Skipped 22 previous similar messages [ 9707.967644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9717.517794] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 9717.520601] Lustre: Skipped 10 previous similar messages [ 9722.336240] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9722.503678] Lustre: server umount lustre-MDT0000 complete [ 9722.638497] LustreError: 6557:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9722.645658] LustreError: 6557:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 136 previous similar messages [ 9740.256210] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9740.260044] LustreError: Skipped 3 previous similar messages [ 9740.453875] LDISKFS-fs (dm-0): recovery complete [ 9740.455880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.500762] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdace15171 [ 9750.506682] Lustre: MGC192.168.206.151@tcp: Connection restored to 192.168.206.151@tcp (at 0@lo) [ 9750.509977] Lustre: Skipped 20 previous similar messages [ 9750.657911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9750.659930] Lustre: Skipped 6 previous similar messages [ 9750.693892] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9750.696784] Lustre: Skipped 6 previous similar messages [ 9750.908202] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9750.911016] Lustre: Skipped 6 previous similar messages [ 9753.148578] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9759.111125] Lustre: lustre-MDT0000: Recovery over after 0:09, of 2 clients 2 recovered and 0 were evicted. [ 9759.113841] Lustre: Skipped 6 previous similar messages [ 9759.139985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6516 to 0x280000401:6561) [ 9759.143696] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6483 to 0x2c0000401:6529) [ 9759.161330] Lustre: 8910:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e951f191d00 x1828185064329344/t335007460574(0) o36->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:698/0 lens 488/3152 e 0 to 0 dl 1743502568 ref 1 fl Interpret:/202/0 rc 0/0 job:'tar.0' uid:0 gid:0 [ 9762.985938] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9765.413719] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9798.652095] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 06:16:34 (1743502594) [ 9925.998220] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 9938.490629] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 9940.122994] Lustre: Failing over lustre-MDT0001 [ 9940.310593] Lustre: server umount lustre-MDT0001 complete [ 9958.058446] LDISKFS-fs (dm-1): recovery complete [ 9958.059911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9958.107515] Lustre: 180807:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [ 9960.694655] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 9964.646766] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3290 to 0x2c0000400:3329) [ 9964.648307] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3323 to 0x280000400:3361) [ 9968.941317] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9971.962286] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10100.680487] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10113.436591] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [10115.196470] Lustre: Failing over lustre-MDT0000 [10115.226205] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [10115.229472] Lustre: Skipped 4 previous similar messages [10121.325307] Lustre: server umount lustre-MDT0000 complete [10138.592418] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743502920/real 1743502920] req@ffff9e95165e0040 x1828185084171264/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743502936 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [10138.602126] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10139.203714] LDISKFS-fs (dm-0): recovery complete [10139.205974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10148.836708] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdacf1fdd4 [10151.265866] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10156.070400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7105 to 0x280000401:7137) [10156.071943] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7073 to 0x2c0000401:7105) [10161.108507] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10164.234920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10174.475636] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 06:22:50 (1743502970) [10301.994754] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10314.834789] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [10316.632675] Lustre: Failing over lustre-MDT0001 [10316.635515] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -19 [10316.638759] LustreError: Skipped 2 previous similar messages [10316.640731] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10316.645868] Lustre: Skipped 10 previous similar messages [10316.649967] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10316.652461] Lustre: Skipped 5 previous similar messages [10316.845195] Lustre: server umount lustre-MDT0001 complete [10323.425452] LustreError: 6558:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10323.430283] LustreError: 6558:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 88 previous similar messages [10334.167495] LDISKFS-fs (dm-1): recovery complete [10334.170087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10334.232539] Lustre: 184450:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [10336.633706] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10339.845107] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3573 to 0x2c0000400:3617) [10339.845301] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3604 to 0x280000400:3649) [10344.187818] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10347.448840] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10476.446527] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10489.175526] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [10491.120752] Lustre: Failing over lustre-MDT0001 [10491.160909] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10491.272729] Lustre: server umount lustre-MDT0001 complete [10508.451488] LDISKFS-fs (dm-1): recovery complete [10508.453382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10508.504212] Lustre: 186164:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [10508.637544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10508.640783] Lustre: Skipped 3 previous similar messages [10508.667403] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [10508.670022] Lustre: Skipped 3 previous similar messages [10509.069557] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [10509.072374] Lustre: Skipped 3 previous similar messages [10510.787753] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10513.894079] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [10513.896107] Lustre: Skipped 15 previous similar messages [10513.910139] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10513.913467] Lustre: Skipped 3 previous similar messages [10513.930719] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3573 to 0x2c0000400:3649) [10513.930803] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3604 to 0x280000400:3681) [10518.614858] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10521.601192] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10536.329363] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 06:28:52 (1743503332) [10545.447823] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [10547.852328] Lustre: DEBUG MARKER: test_70f failing OST 1 times [10549.216395] Lustre: Failing over lustre-OST0000 [10549.281726] Lustre: server umount lustre-OST0000 complete [10566.362110] LDISKFS-fs (dm-2): recovery complete [10566.364188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10566.398452] Lustre: 188012:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [10566.488080] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10570.029226] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10577.408163] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10580.291652] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10593.386744] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [10595.719513] Lustre: DEBUG MARKER: test_70f failing OST 2 times [10597.105844] Lustre: Failing over lustre-OST0000 [10597.150430] Lustre: server umount lustre-OST0000 complete [10614.440918] LDISKFS-fs (dm-2): recovery complete [10614.443073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10614.488757] Lustre: 189758:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [10614.576748] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10618.600592] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10627.247415] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10630.174981] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10642.743220] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [10645.187754] Lustre: DEBUG MARKER: test_70f failing OST 3 times [10646.579866] Lustre: Failing over lustre-OST0000 [10646.622155] Lustre: server umount lustre-OST0000 complete [10663.778570] LDISKFS-fs (dm-2): recovery complete [10663.779772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10663.814444] Lustre: 191505:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [10663.897401] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [10667.424220] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10676.352571] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10679.698713] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [10692.641826] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 06:31:28 (1743503488) [10819.995309] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [10825.644381] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10838.218426] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [10839.940175] Lustre: Failing over lustre-MDT0001 [10839.962579] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [10840.206156] Lustre: server umount lustre-MDT0001 complete [10842.373560] Lustre: Failing over lustre-MDT0000 [10842.662542] Lustre: server umount lustre-MDT0000 complete [10859.488194] Lustre: 3682:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743503641/real 1743503641] req@ffff9e9508f6bf80 x1828185091837056/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743503657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [10859.488390] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10859.496013] Lustre: 3682:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [10859.503959] LustreError: Skipped 1 previous similar message [10863.938121] LDISKFS-fs (dm-1): recovery complete [10863.941448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10864.157941] LustreError: 194585:0:(llog.c:1515:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [10864.163030] Lustre: 194585:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [10864.211029] LDISKFS-fs (dm-0): recovery complete [10864.212905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10869.729512] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e951cdeb400 x1828185091840768/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [10873.067493] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10873.360319] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [10876.090357] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3604 to 0x280000400:3713) [10876.090455] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3573 to 0x2c0000400:3681) [10881.068823] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8706 to 0x2c0000401:8737) [10881.069408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8738 to 0x280000401:8769) [10886.567996] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10889.598576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10892.319243] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11021.058111] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [11026.676635] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11039.506138] Lustre: DEBUG MARKER: fail mds2 mds1 2 times [11041.126928] Lustre: Failing over lustre-MDT0001 [11041.129933] LustreError: lustre-MDT0001-osp-MDT0000: operation dt_index_read to node 0@lo failed: rc = -19 [11041.129945] LustreError: Skipped 6 previous similar messages [11041.129970] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11041.129977] Lustre: Skipped 16 previous similar messages [11041.380248] Lustre: server umount lustre-MDT0001 complete [11043.562654] Lustre: Failing over lustre-MDT0000 [11043.637634] LustreError: 196719:0:(ldlm_resource.c:983:ldlm_resource_complain()) lustre-MDT0001-osp-MDT0000: namespace resource [0x240003e98:0x8ea:0x0].0x0 (ffff9e9520dbdd80) refcount nonzero (1) after lock cleanup; forcing cleanup. [11043.862453] Lustre: server umount lustre-MDT0000 complete [11064.241107] LDISKFS-fs (dm-1): recovery complete [11064.243528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11064.441266] LustreError: 197809:0:(llog.c:1515:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [11064.447198] Lustre: 197809:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [11064.520507] LDISKFS-fs (dm-0): recovery complete [11064.522277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11071.969215] LustreError: 197826:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11071.970587] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdad0d2a8b [11071.974307] LustreError: 197826:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 69 previous similar messages [11075.066310] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11075.362789] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11078.332696] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3604 to 0x280000400:3745) [11078.332791] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3573 to 0x2c0000400:3713) [11087.615371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8738 to 0x280000401:8801) [11087.615845] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8706 to 0x2c0000401:8769) [11093.224595] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11095.728848] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11098.062113] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11107.552743] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 06:38:23 (1743503903) [11112.373883] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11113.940270] Lustre: Failing over lustre-MDT0000 [11114.081247] Lustre: server umount lustre-MDT0000 complete [11131.202297] LDISKFS-fs (dm-0): recovery complete [11131.203429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11140.067621] Lustre: MGC192.168.206.151@tcp: Connection restored to 192.168.206.151@tcp (at 0@lo) [11140.070148] Lustre: Skipped 21 previous similar messages [11140.175927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11140.179415] Lustre: Skipped 4 previous similar messages [11140.213940] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11140.216374] Lustre: Skipped 7 previous similar messages [11141.930324] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11141.932821] Lustre: Skipped 7 previous similar messages [11141.936037] Lustre: *** cfs_fail_loc=302, val=2147483648*** [11142.365652] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11158.290274] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:52 [11158.305310] Lustre: 197825:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e9509b06200 x1828185085099520/t343597400568(343597400568) o101->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:587/0 lens 520/3488 e 0 to 0 dl 1743503967 ref 1 fl Interpret:/206/0 rc 0/0 job:'lfs.0' uid:0 gid:0 [11158.328520] Lustre: lustre-MDT0000: Recovery over after 0:17, of 2 clients 2 recovered and 0 were evicted. [11158.332264] Lustre: Skipped 7 previous similar messages [11158.346292] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:8833) [11158.346605] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8706 to 0x2c0000401:8801) [11161.769335] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11163.622856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11172.658064] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 06:39:28 (1743503968) [11177.338965] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11178.765053] Lustre: Failing over lustre-MDT0000 [11178.923488] Lustre: server umount lustre-MDT0000 complete [11196.285354] LDISKFS-fs (dm-0): recovery complete [11196.286587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11196.387686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [11196.389364] Lustre: Skipped 2 previous similar messages [11198.244230] Lustre: *** cfs_fail_loc=157, val=2147483648*** [11198.246328] LustreError: 197848:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e951bfef340 x1828185085099520/t343597400568(343597400568) o101->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:627/0 lens 520/664 e 0 to 0 dl 1743504007 ref 1 fl Interpret:/204/0 rc 301/0 job:'lfs.0' uid:0 gid:0 [11198.579561] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11213.600927] Lustre: lustre-MDT0000: Client c154a474-28d3-40b8-9d00-0def439c1563 (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [11213.605377] Lustre: 197824:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e950adb4540 x1828185085099520/t343597400568(343597400568) o101->c154a474-28d3-40b8-9d00-0def439c1563@192.168.206.51@tcp:642/0 lens 520/3488 e 0 to 0 dl 1743504022 ref 1 fl Interpret:/206/0 rc 0/0 job:'lfs.0' uid:0 gid:0 [11213.636718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:8865) [11213.636750] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8803 to 0x2c0000401:8833) [11217.261818] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11219.344637] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11228.815877] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 06:40:24 (1743504024) [11232.052363] Lustre: Failing over lustre-OST0000 [11232.105944] Lustre: server umount lustre-OST0000 complete [11234.140990] Lustre: Failing over lustre-MDT0000 [11234.273203] Lustre: server umount lustre-MDT0000 complete [11249.979303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.661162] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11254.125980] Lustre: lustre-MDT0000: Denying connection for new client b2b994f9-80ee-4adf-b1f2-9ec74d18207f (at 192.168.206.51@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [11254.129849] Lustre: Skipped 13 previous similar messages [11255.287260] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8803 to 0x2c0000401:8865) [11262.629894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11262.672060] Lustre: 204129:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [11264.108851] Lustre: lustre-OST0000: Denying connection for new client b2b994f9-80ee-4adf-b1f2-9ec74d18207f (at 192.168.206.51@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [11264.115896] Lustre: Skipped 1 previous similar message [11264.428128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:8897) [11265.996722] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11276.293762] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 06:41:12 (1743504072) [11277.255662] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [11277.257680] LustreError: 198652:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e9626c10600 x1828185098338688/t365072220167(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:706/0 lens 1312/4320 e 0 to 0 dl 1743504086 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 [11281.159192] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11282.260757] Lustre: Failing over lustre-MDT0000 [11282.528222] Lustre: server umount lustre-MDT0000 complete [11300.212971] LDISKFS-fs (dm-0): recovery complete [11300.215359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11311.572897] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11314.178861] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:8897) [11314.179707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:8929) [11317.983638] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11319.799542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11329.825354] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 06:42:05 (1743504125) [11330.712667] LustreError: 197829:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e951be8a2c0 x1828185098378240/t369367187469(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:4/0 lens 2520/4320 e 0 to 0 dl 1743504139 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 [11334.902469] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11339.077405] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [11340.066204] Lustre: Failing over lustre-MDT0001 [11345.995620] Lustre: server umount lustre-MDT0001 complete [11362.976748] LDISKFS-fs (dm-1): recovery complete [11362.977880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11363.020016] Lustre: 208184:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [11365.478655] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11368.445680] Lustre: 202406:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e95060ed680 x1828185092173568/t38654707382(0) o36->b2b994f9-80ee-4adf-b1f2-9ec74d18207f@192.168.206.51@tcp:68/0 lens 560/2880 e 0 to 0 dl 1743504203 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 [11368.447741] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3724 to 0x2c0000400:3745) [11368.447755] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3756 to 0x280000400:3777) [11372.126040] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11374.434676] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11385.372164] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 06:43:01 (1743504181) [11386.377440] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [11386.379924] Lustre: Skipped 1 previous similar message [11390.734852] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11395.158350] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [11396.237715] Lustre: Failing over lustre-MDT0000 [11396.502790] Lustre: server umount lustre-MDT0000 complete [11413.958810] LDISKFS-fs (dm-0): recovery complete [11413.960454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11416.434189] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11419.677339] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:8929) [11419.677406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:8961) [11422.750310] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11424.943665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11428.244600] Lustre: Failing over lustre-MDT0001 [11428.346252] Lustre: server umount lustre-MDT0001 complete [11445.887890] LDISKFS-fs (dm-1): recovery complete [11445.889236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11445.934088] Lustre: 211653:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [11448.304016] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11451.389288] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3756 to 0x2c0000400:3777) [11451.389291] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3788 to 0x280000400:3809) [11454.857308] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11456.775840] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11468.081914] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 06:44:23 (1743504263) [11469.267977] LustreError: 198652:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e961fa20bc0 x1828185098475648/t373662154768(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:143/0 lens 2520/4320 e 0 to 0 dl 1743504278 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 [11469.273894] LustreError: 198652:0:(ldlm_lib.c:3251:target_send_reply_msg()) Skipped 1 previous similar message [11477.047527] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11481.889526] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [11483.166962] Lustre: Failing over lustre-MDT0000 [11483.502174] Lustre: server umount lustre-MDT0000 complete [11485.664093] Lustre: 211660:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743504267/real 1743504267] req@ffff9e961fa26200 x1828185098475648/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1743504283 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 [11485.671633] Lustre: 211660:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 34 previous similar messages [11485.866561] LustreError: 6542:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504283 with bad export cookie 7018337741252199531 [11485.866869] Lustre: Failing over lustre-MDT0001 [11485.867993] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11485.868040] LustreError: Skipped 6 previous similar messages [11485.871229] LustreError: 6542:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [11491.809661] LustreError: 213481:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9e9626c9dc40 x1828185098486784/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [11491.815422] LustreError: 213481:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [11491.984407] Lustre: server umount lustre-MDT0001 complete [11514.241734] LDISKFS-fs (dm-0): recovery complete [11514.244449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11514.417213] LDISKFS-fs (dm-1): recovery complete [11514.418933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11530.720662] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9626c99d00 x1828185098488192/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [11534.540589] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11535.165143] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11537.724242] Lustre: 214601:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e951cf5d680 x1828185092246912/t47244640326(0) o36->b2b994f9-80ee-4adf-b1f2-9ec74d18207f@192.168.206.51@tcp:236/0 lens 560/2880 e 0 to 0 dl 1743504371 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 [11537.726359] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3788 to 0x2c0000400:3809) [11537.727331] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3820 to 0x280000400:3841) [11554.097996] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:8961) [11554.098679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:8993) [11557.867915] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11560.022025] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11561.893230] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11571.565741] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 06:46:07 (1743504367) [11572.562577] Lustre: *** cfs_fail_loc=119, val=2147483648*** [11572.564827] Lustre: Skipped 1 previous similar message [11579.866929] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11581.018765] Lustre: Failing over lustre-MDT0000 [11581.290369] Lustre: server umount lustre-MDT0000 complete [11588.878128] Lustre: lustre-MDT0001: Client b2b994f9-80ee-4adf-b1f2-9ec74d18207f (at 192.168.206.51@tcp) reconnecting [11588.881196] Lustre: Skipped 2 previous similar messages [11588.884408] Lustre: 215331:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e9521850040 x1828185092284032/t51539607622(0) o36->b2b994f9-80ee-4adf-b1f2-9ec74d18207f@192.168.206.51@tcp:262/0 lens 560/2880 e 0 to 0 dl 1743504397 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 [11598.779301] LDISKFS-fs (dm-0): recovery complete [11598.781424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11601.388680] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11604.495296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:9025) [11604.495517] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:8993) [11607.590435] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11609.522394] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11618.098280] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 06:46:54 (1743504414) [11619.115228] LustreError: 214601:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e9521b11740 x1828185092316544/t51539607693(0) o36->b2b994f9-80ee-4adf-b1f2-9ec74d18207f@192.168.206.51@tcp:293/0 lens 560/448 e 0 to 0 dl 1743504428 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 [11619.125297] LustreError: 214601:0:(ldlm_lib.c:3251:target_send_reply_msg()) Skipped 1 previous similar message [11623.927993] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [11625.065598] Lustre: Failing over lustre-MDT0001 [11625.180284] Lustre: server umount lustre-MDT0001 complete [11642.624946] LDISKFS-fs (dm-1): recovery complete [11642.626175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11642.680116] Lustre: 218758:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [11642.682920] Lustre: 218758:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [11644.917777] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11647.993390] Lustre: 214600:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e951e91a880 x1828185092316544/t51539607693(0) o36->b2b994f9-80ee-4adf-b1f2-9ec74d18207f@192.168.206.51@tcp:322/0 lens 560/2880 e 0 to 0 dl 1743504457 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 [11647.996215] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3862 to 0x280000400:3905) [11647.996410] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3830 to 0x2c0000400:3873) [11651.163392] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11652.977849] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11662.549990] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 06:47:38 (1743504458) [11670.697372] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11674.883973] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [11675.995030] Lustre: Failing over lustre-MDT0000 [11676.185700] Lustre: server umount lustre-MDT0000 complete [11678.690566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11678.690597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11678.691934] LustreError: 215501:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11678.691943] LustreError: 215501:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 190 previous similar messages [11678.693693] Lustre: Skipped 46 previous similar messages [11678.696042] LustreError: Skipped 9 previous similar messages [11680.019178] Lustre: lustre-MDT0001: Client b2b994f9-80ee-4adf-b1f2-9ec74d18207f (at 192.168.206.51@tcp) reconnecting [11680.023655] Lustre: 215501:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e961f90d0c0 x1828185092347776/t55834574918(0) o36->b2b994f9-80ee-4adf-b1f2-9ec74d18207f@192.168.206.51@tcp:354/0 lens 560/2880 e 0 to 0 dl 1743504489 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 [11693.852240] LDISKFS-fs (dm-0): recovery complete [11693.854056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11696.305816] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11699.208619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:9057) [11699.208764] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9025) [11702.361766] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11704.503229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11707.778791] Lustre: Failing over lustre-MDT0001 [11707.871648] Lustre: server umount lustre-MDT0001 complete [11725.276939] LDISKFS-fs (dm-1): recovery complete [11725.278923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11727.708358] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11730.947903] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3916 to 0x280000400:3937) [11730.947925] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3884 to 0x2c0000400:3905) [11734.196280] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11736.293943] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11746.307717] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 06:49:02 (1743504542) [11754.461407] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11758.770802] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [11759.867922] Lustre: Failing over lustre-MDT0000 [11760.126497] Lustre: server umount lustre-MDT0000 complete [11762.154687] LustreError: 9452:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504560 with bad export cookie 7018337741252213272 [11762.155733] Lustre: Failing over lustre-MDT0001 [11762.158316] LustreError: 9452:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [11762.304784] Lustre: server umount lustre-MDT0001 complete [11782.055328] LDISKFS-fs (dm-0): recovery complete [11782.057466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11782.239756] LDISKFS-fs (dm-1): recovery complete [11782.242022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11787.232697] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9507dec540 x1828185098678144/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [11787.455784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11787.458458] Lustre: Skipped 13 previous similar messages [11787.484303] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11787.486740] Lustre: Skipped 13 previous similar messages [11789.070747] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [11789.074475] Lustre: Skipped 13 previous similar messages [11790.881173] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11791.394796] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11791.975677] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [11791.978803] Lustre: Skipped 48 previous similar messages [11793.019972] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11793.022257] Lustre: Skipped 13 previous similar messages [11793.033366] Lustre: 225789:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e9509b050c0 x1828185092388992/t60129542214(0) o36->b2b994f9-80ee-4adf-b1f2-9ec74d18207f@192.168.206.51@tcp:467/0 lens 560/2880 e 0 to 0 dl 1743504602 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 [11793.036963] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3916 to 0x2c0000400:3937) [11793.037024] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3948 to 0x280000400:3969) [11799.357121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:9089) [11799.358431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9057) [11803.000029] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11804.831071] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11806.947958] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11816.449856] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 06:50:12 (1743504612) [11821.831037] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [11822.920817] Lustre: Failing over lustre-MDT0001 [11822.925994] LustreError: 226826:0:(ldlm_resource.c:983:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029c11:0x17:0x0].0x0 (ffff9e961f5b0380) refcount nonzero (2) after lock cleanup; forcing cleanup. [11822.937804] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [11822.940603] Lustre: Skipped 9 previous similar messages [11828.804561] Lustre: server umount lustre-MDT0001 complete [11846.787104] LDISKFS-fs (dm-1): recovery complete [11846.789476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11849.455975] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11852.290253] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3980 to 0x280000400:4001) [11852.290803] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3948 to 0x2c0000400:3969) [11856.375473] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11858.578619] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11868.807085] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 06:51:05 (1743504665) [11869.811598] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [11869.813582] Lustre: Skipped 4 previous similar messages [11874.159305] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11875.210097] Lustre: Failing over lustre-MDT0000 [11875.492175] Lustre: server umount lustre-MDT0000 complete [11892.543702] LDISKFS-fs (dm-0): recovery complete [11892.545695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11894.777439] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11897.864142] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9089) [11897.864280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:9121) [11900.922344] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11902.805928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11911.173446] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 06:51:47 (1743504707) [11912.059563] LustreError: 8441:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e9515d9e200 x1828185098784896/t395136991244(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:586/0 lens 1744/4320 e 0 to 0 dl 1743504721 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 [11912.066429] LustreError: 8441:0:(ldlm_lib.c:3251:target_send_reply_msg()) Skipped 4 previous similar messages [11916.129669] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11920.402538] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [11921.521871] Lustre: Failing over lustre-MDT0000 [11921.738855] Lustre: server umount lustre-MDT0000 complete [11939.077451] LDISKFS-fs (dm-0): recovery complete [11939.079470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11941.403994] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11944.455380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:9153) [11944.455388] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9121) [11947.343492] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11949.308421] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11952.543206] Lustre: Failing over lustre-MDT0001 [11952.645376] Lustre: server umount lustre-MDT0001 complete [11969.586088] LDISKFS-fs (dm-1): recovery complete [11969.587416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11969.636872] Lustre: 232777:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [11969.641325] Lustre: 232777:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 3 previous similar messages [11972.101507] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [11975.167950] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3980 to 0x280000400:4033) [11975.167960] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3948 to 0x2c0000400:4001) [11978.057936] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11979.731871] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [11988.497750] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 06:53:04 (1743504784) [11993.588141] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [11998.275382] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [11999.449689] Lustre: Failing over lustre-MDT0000 [11999.746034] Lustre: server umount lustre-MDT0000 complete [12001.962251] LustreError: 6543:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743504800 with bad export cookie 7018337741252223674 [12001.963359] Lustre: Failing over lustre-MDT0001 [12001.968140] LustreError: 6543:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [12001.972872] LustreError: 234602:0:(ldlm_resource.c:983:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029c11:0x1f:0x0].0xf7117594 (ffff9e961f5b1d80) refcount nonzero (1) after lock cleanup; forcing cleanup. [12001.980427] LustreError: 234602:0:(ldlm_resource.c:983:ldlm_resource_complain()) Skipped 1 previous similar message [12007.905845] LustreError: 234602:0:(client.c:1282:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9e950adb2e40 x1828185098846976/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 [12007.911515] LustreError: 234602:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [12008.092690] Lustre: server umount lustre-MDT0001 complete [12030.086653] LDISKFS-fs (dm-1): recovery complete [12030.088847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12030.092896] LDISKFS-fs (dm-0): recovery complete [12030.095252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12030.224415] LustreError: 235701:0:(llog.c:1515:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [12030.226523] Lustre: 235701:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [12047.842154] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdad0e6c8b [12047.844276] Lustre: Skipped 2 previous similar messages [12051.163297] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12051.639550] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12054.172770] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3980 to 0x280000400:4065) [12054.173571] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3948 to 0x2c0000400:4033) [12054.184379] Lustre: 235712:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e961f920bc0 x1828185092472960/t73014444034(0) o36->b2b994f9-80ee-4adf-b1f2-9ec74d18207f@192.168.206.51@tcp:753/0 lens 496/2888 e 0 to 0 dl 1743504888 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 [12054.192233] Lustre: 235712:0:(mdt_recovery.c:128:mdt_req_from_lrd()) Skipped 1 previous similar message [12054.240299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:9185) [12054.240323] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9153) [12057.963843] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12059.744455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12061.345657] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [12070.564230] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 06:54:26 (1743504866) [12076.298617] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [12077.431582] Lustre: Failing over lustre-MDT0000 [12077.612601] Lustre: server umount lustre-MDT0000 complete [12087.053886] Lustre: lustre-MDT0001: Client b2b994f9-80ee-4adf-b1f2-9ec74d18207f (at 192.168.206.51@tcp) reconnecting [12095.248858] LDISKFS-fs (dm-0): recovery complete [12095.250888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12095.316629] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12095.320872] LustreError: Skipped 6 previous similar messages [12097.831406] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12100.621954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:9217) [12100.622041] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9185) [12104.077876] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12106.152540] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12116.047846] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 06:55:12 (1743504912) [12121.549522] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [12122.681395] Lustre: Failing over lustre-MDT0001 [12122.791284] Lustre: server umount lustre-MDT0001 complete [12140.388737] LDISKFS-fs (dm-1): recovery complete [12140.389936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12143.000280] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12145.661056] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3980 to 0x280000400:4097) [12145.661198] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3948 to 0x2c0000400:4065) [12149.205994] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12151.458369] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [12161.207771] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 06:55:57 (1743504957) [12166.714874] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [12171.300308] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [12172.480208] Lustre: Failing over lustre-MDT0000 [12172.707092] Lustre: server umount lustre-MDT0000 complete [12190.547877] LDISKFS-fs (dm-0): recovery complete [12190.549025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12193.022659] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12195.853223] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9217) [12195.853288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:9249) [12199.213658] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12201.312623] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12204.672073] Lustre: Failing over lustre-MDT0001 [12204.777648] Lustre: server umount lustre-MDT0001 complete [12222.203274] LDISKFS-fs (dm-1): recovery complete [12222.205305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12224.744434] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12227.581496] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3948 to 0x2c0000400:4097) [12227.581496] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3980 to 0x280000400:4129) [12230.969868] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12233.166853] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [12242.924888] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 06:57:18 (1743505038) [12248.104513] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [12252.327053] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [12253.434246] Lustre: Failing over lustre-MDT0000 [12253.594422] Lustre: server umount lustre-MDT0000 complete [12255.733183] LustreError: 6543:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505053 with bad export cookie 7018337741252231206 [12255.734523] Lustre: Failing over lustre-MDT0001 [12255.735895] LustreError: 6543:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [12255.920334] Lustre: server umount lustre-MDT0001 complete [12274.144137] Lustre: 3680:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743505056/real 1743505056] req@ffff9e951cf58040 x1828185099011072/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743505072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [12274.155754] Lustre: 3680:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 16 previous similar messages [12277.770428] LDISKFS-fs (dm-1): recovery complete [12277.772510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12277.944914] LDISKFS-fs (dm-0): recovery complete [12277.946760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12277.947765] LustreError: 246276:0:(llog.c:1515:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [12277.956768] Lustre: 246276:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [12280.032170] LustreError: 246301:0:(import.c:333:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [12280.036478] LustreError: 246301:0:(import.c:357:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9e9507dec540 x1828185099012608/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1743505078 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [12280.046389] LustreError: 246301:0:(import.c:367:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [12280.288572] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9507dee7c0 x1828185099014400/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [12280.381813] LustreError: 175905:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12280.387027] LustreError: 175905:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 192 previous similar messages [12280.679251] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [12280.681066] LustreError: Skipped 12 previous similar messages [12284.434603] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12284.820738] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12285.660445] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3948 to 0x2c0000400:4129) [12285.662038] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3980 to 0x280000400:4161) [12285.683608] Lustre: 246305:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e9636ed5c40 x1828185092538880/t85899345922(0) o36->b2b994f9-80ee-4adf-b1f2-9ec74d18207f@192.168.206.51@tcp:204/0 lens 496/2888 e 0 to 0 dl 1743505094 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 [12285.692882] Lustre: 246305:0:(mdt_recovery.c:128:mdt_req_from_lrd()) Skipped 3 previous similar messages [12295.987638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8803 to 0x280000401:9281) [12295.987965] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8867 to 0x2c0000401:9249) [12299.891803] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12301.938402] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12303.863551] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [12313.256703] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 06:58:29 (1743505109) [12314.513043] Lustre: 247547:0:(genops.c:1678:obd_export_evict_by_uuid()) lustre-MDT0000: evicting b2b994f9-80ee-4adf-b1f2-9ec74d18207f at adminstrative request [12324.553651] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 06:58:40 (1743505120) [12328.682439] Lustre: Failing over lustre-MDT0000 [12328.923407] Lustre: server umount lustre-MDT0000 complete [12332.006789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12332.011398] Lustre: Skipped 48 previous similar messages [12344.498974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12347.102650] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12350.013946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9332 to 0x280000401:9377) [12350.014691] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9301 to 0x2c0000401:9345) [12353.554888] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12355.189567] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12363.831214] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 06:59:20 (1743505160) [12371.623836] Lustre: Failing over lustre-OST0000 [12373.715853] Lustre: server umount lustre-OST0000 complete [12388.813543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12388.926673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12388.928440] Lustre: Skipped 14 previous similar messages [12388.932742] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [12388.935723] Lustre: Skipped 14 previous similar messages [12390.215447] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [12390.219643] Lustre: Skipped 14 previous similar messages [12392.440666] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12398.541454] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12400.272377] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [12409.857381] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 07:00:05 (1743505205) [12412.965218] Lustre: Failing over lustre-MDT0000 [12413.208675] Lustre: server umount lustre-MDT0000 complete [12418.144868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12420.845475] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12422.484098] Lustre: lustre-MDT0000: Denying connection for new client 97b58810-b90d-40c2-919c-0b7624a0942a (at 192.168.206.51@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [12423.655973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [12423.658612] Lustre: Skipped 52 previous similar messages [12423.666937] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [12423.670039] Lustre: Skipped 15 previous similar messages [12423.686594] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9301 to 0x2c0000401:9377) [12423.686610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9332 to 0x280000401:9409) [12436.420728] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 07:00:32 (1743505232) [12442.206272] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [12444.096856] Lustre: Failing over lustre-OST0000 [12444.146106] Lustre: server umount lustre-OST0000 complete [12462.249372] LDISKFS-fs (dm-2): recovery complete [12462.251622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12466.040602] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12471.903360] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12473.848218] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [12482.785468] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 07:01:19 (1743505279) [12487.716720] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [12490.371755] Lustre: Failing over lustre-OST0000 [12490.409639] Lustre: server umount lustre-OST0000 complete [12507.792345] LDISKFS-fs (dm-2): recovery complete [12507.794429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12507.841595] Lustre: 254486:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [12507.845262] Lustre: 254486:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 4 previous similar messages [12509.286743] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.206.51@tcp inode [0x200030971:0x5:0x0] object 0x280000401:9411 extent [0-4194303]: client csum b6c61798, server csum 647555df [12511.437168] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12517.809674] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12519.819601] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [12530.474531] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 07:02:06 (1743505326) [12536.133110] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [12541.011364] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [12545.093273] Lustre: Failing over lustre-MDT0000 [12545.296401] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [12545.298258] Lustre: Skipped 13 previous similar messages [12545.374144] Lustre: server umount lustre-MDT0000 complete [12547.635890] LustreError: 6543:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505345 with bad export cookie 7018337741252258086 [12547.639289] Lustre: Failing over lustre-OST0000 [12547.640972] LustreError: 6543:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 2 previous similar messages [12547.685327] Lustre: server umount lustre-OST0000 complete [12565.775156] LDISKFS-fs (dm-0): recovery complete [12565.776696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12575.436653] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12578.464827] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9301 to 0x2c0000401:9409) [12593.897635] LDISKFS-fs (dm-2): recovery complete [12593.898938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12597.281559] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12610.094788] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 07:03:26 (1743505406) [12617.857609] Lustre: Failing over lustre-OST0000 [12617.919047] Lustre: server umount lustre-OST0000 complete [12620.130192] Lustre: Failing over lustre-MDT0000 [12620.483469] Lustre: server umount lustre-MDT0000 complete [12635.903414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12638.228621] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12641.277492] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9301 to 0x2c0000401:9441) [12644.615241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12648.034955] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12649.464510] Lustre: lustre-OST0000: Denying connection for new client 7d8d0c82-6141-4101-872b-23cb3da12043 (at 192.168.206.51@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [12649.470252] Lustre: Skipped 1 previous similar message [12654.865749] Lustre: lustre-OST0000: Denying connection for new client 7d8d0c82-6141-4101-872b-23cb3da12043 (at 192.168.206.51@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [12654.872803] Lustre: Skipped 1 previous similar message [12665.101362] Lustre: lustre-OST0000: Denying connection for new client 7d8d0c82-6141-4101-872b-23cb3da12043 (at 192.168.206.51@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:50 [12665.106847] Lustre: Skipped 1 previous similar message [12685.586524] Lustre: lustre-OST0000: Denying connection for new client 7d8d0c82-6141-4101-872b-23cb3da12043 (at 192.168.206.51@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:30 [12685.591631] Lustre: Skipped 3 previous similar messages [12716.000119] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [12716.002269] Lustre: 260559:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 97b58810-b90d-40c2-919c-0b7624a0942a@ [12716.005305] Lustre: 260559:0:(genops.c:1508:class_disconnect_stale_exports()) Skipped 1 previous similar message [12716.007357] Lustre: lustre-OST0000: disconnecting 1 stale clients [12716.021055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9460 to 0x280000401:9481) [12718.224401] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 59 sec [12728.060775] Lustre: DEBUG MARKER: free_before: 7646296 free_after: 7646296 [12736.940153] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 07:05:32 (1743505532) [12740.295394] Lustre: Failing over lustre-OST0000 [12740.346576] Lustre: server umount lustre-OST0000 complete [12756.825422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12760.545049] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12773.083968] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 07:06:08 (1743505568) [12775.924215] Lustre: Failing over lustre-OST0000 [12775.968854] Lustre: server umount lustre-OST0000 complete [12791.715910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12793.593703] LustreError: 263628:0:(ldlm_lib.c:2808:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [12793.596292] LustreError: 263628:0:(ldlm_lib.c:2808:target_recovery_thread()) Skipped 102 previous similar messages [12795.470945] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12799.456183] Lustre: *** cfs_fail_loc=715, val=40*** [12809.185596] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:53 [12809.190205] Lustre: Skipped 1 previous similar message [12809.997416] Lustre: lustre-OST0000: Client 7d8d0c82-6141-4101-872b-23cb3da12043 (at 192.168.206.51@tcp) reconnected, waiting for 3 clients in recovery for 0:53 [12815.328280] Lustre: *** cfs_fail_loc=715, val=40*** [12815.329431] Lustre: Skipped 2 previous similar messages [12816.352141] Lustre: *** cfs_fail_loc=715, val=40*** [12816.354034] Lustre: Skipped 1 previous similar message [12825.569734] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 0:37 [12825.574612] Lustre: Skipped 1 previous similar message [12831.712248] Lustre: *** cfs_fail_loc=715, val=40*** [12833.656199] LustreError: 263628:0:(ldlm_lib.c:2808:target_recovery_thread()) cfs_fail_timeout id 715 awake [12833.658380] LustreError: 263628:0:(ldlm_lib.c:2808:target_recovery_thread()) Skipped 104 previous similar messages [12837.926740] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12840.452575] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [12850.320849] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 07:07:26 (1743505646) [12853.555443] Lustre: Failing over lustre-MDT0000 [12853.941399] Lustre: server umount lustre-MDT0000 complete [12869.806639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12869.873471] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12869.878246] LustreError: Skipped 6 previous similar messages [12872.578422] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12875.246461] LustreError: 265067:0:(ldlm_lib.c:2808:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [12878.304137] Lustre: *** cfs_fail_loc=715, val=80*** [12878.306097] Lustre: Skipped 2 previous similar messages [12888.333345] Lustre: lustre-MDT0000: Client 7d8d0c82-6141-4101-872b-23cb3da12043 (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:52 [12888.338837] Lustre: Skipped 1 previous similar message [12891.616150] Lustre: 3678:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743505673/real 1743505673] req@ffff9e960796a880 x1828185099380992/t0(0) o400->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 1 dl 1743505689 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 [12891.624365] Lustre: 3678:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [12891.627851] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [12894.688207] Lustre: *** cfs_fail_loc=715, val=80*** [12894.690125] Lustre: Skipped 1 previous similar message [12903.693942] Lustre: lustre-MDT0000: Client 7d8d0c82-6141-4101-872b-23cb3da12043 (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [12906.979401] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [12913.120190] Lustre: *** cfs_fail_loc=715, val=80*** [12913.122418] Lustre: Skipped 2 previous similar messages [12920.077440] Lustre: lustre-MDT0000: Client 7d8d0c82-6141-4101-872b-23cb3da12043 (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:20 [12923.360792] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [12936.461369] Lustre: lustre-MDT0000: Client 7d8d0c82-6141-4101-872b-23cb3da12043 (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:04 [12939.744735] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [12945.888147] Lustre: *** cfs_fail_loc=715, val=80*** [12945.890375] Lustre: Skipped 3 previous similar messages [12952.845591] Lustre: lustre-MDT0000: Recovery already passed deadline 0:11. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [12955.106159] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [12955.328130] LustreError: 265067:0:(ldlm_lib.c:2808:target_recovery_thread()) cfs_fail_timeout id 715 awake [12955.334489] Lustre: 265067:0:(ldlm_lib.c:2854:target_recovery_thread()) too long recovery - read logs [12955.338452] LustreError: dumping log to /tmp/lustre-log.1743505753.265067 [12955.444909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9495 to 0x280000401:9513) [12955.445051] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9454 to 0x2c0000401:9473) [12959.206359] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12961.222406] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12971.216745] Lustre: DEBUG MARKER: == replay-single test 100a: DNE: create striped dir, drop update rep from MDT1, fail MDT1 ========================================================== 07:09:27 (1743505767) [12972.463697] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [12972.465436] Lustre: Skipped 6 previous similar messages [12972.467166] LustreError: 162827:0:(ldlm_lib.c:3251:target_send_reply_msg()) @@@ dropping reply req@ffff9e9508dc9740 x1828185099425920/t90194313735(0) o1000->lustre-MDT0000-mdtlov_UUID@0@lo:136/0 lens 1312/4320 e 0 to 0 dl 1743505781 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up1-0.0' uid:0 gid:0 [12972.475502] LustreError: 162827:0:(ldlm_lib.c:3251:target_send_reply_msg()) Skipped 5 previous similar messages [12973.509112] Lustre: Failing over lustre-MDT0001 [12973.625939] Lustre: server umount lustre-MDT0001 complete [12975.586252] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12975.588341] LustreError: 246305:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12975.592485] Lustre: Skipped 31 previous similar messages [12975.597473] LustreError: 246305:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 158 previous similar messages [12989.506925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12989.714647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12989.718471] Lustre: Skipped 10 previous similar messages [12989.745553] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [12989.747389] Lustre: Skipped 10 previous similar messages [12991.531298] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [12991.534469] Lustre: Skipped 10 previous similar messages [12992.409939] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [12995.071257] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3948 to 0x2c0000400:4161) [12995.071751] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4262 to 0x280000400:4289) [12999.031772] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13001.401078] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [13012.424535] Lustre: DEBUG MARKER: == replay-single test 100b: DNE: create striped dir, fail MDT0 ========================================================== 07:10:08 (1743505808) [13014.486298] Lustre: Failing over lustre-MDT0000 [13014.726965] Lustre: server umount lustre-MDT0000 complete [13015.523584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13015.526944] LustreError: Skipped 8 previous similar messages [13029.843329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13032.331283] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13035.491691] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.206.151@tcp (at 0@lo) [13035.493482] Lustre: Skipped 29 previous similar messages [13035.501566] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [13035.504157] Lustre: Skipped 10 previous similar messages [13035.504493] Lustre: 247560:0:(mdt_recovery.c:128:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e963708f900 x1828185093202688/t438086664253(0) o36->7d8d0c82-6141-4101-872b-23cb3da12043@192.168.206.51@tcp:199/0 lens 560/2880 e 0 to 0 dl 1743505844 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 [13035.518763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9518 to 0x280000401:9545) [13035.518867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9478 to 0x2c0000401:9505) [13038.486609] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13040.575897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13049.927021] Lustre: DEBUG MARKER: == replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 ========================================================== 07:10:45 (1743505845) [13054.878799] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [13056.009931] Lustre: Failing over lustre-MDT0001 [13056.108329] Lustre: server umount lustre-MDT0001 complete [13063.489924] LDISKFS-fs (dm-1): recovery complete [13063.491154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13063.737663] Lustre: lustre-MDT0001: Aborting MDT recovery [13063.751595] LustreError: 269469:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0000-osp-MDT0001: get update log duration 0, retries 0, failed: rc = -108 [13065.945778] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13068.820802] Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] [13068.827275] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] [13068.861308] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4302 to 0x280000400:4321) [13068.861483] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4174 to 0x2c0000400:4193) [13079.403582] Lustre: Failing over lustre-MDT0001 [13079.618459] Lustre: server umount lustre-MDT0001 complete [13094.944592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13097.418582] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13100.541248] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4198 to 0x2c0000400:4225) [13100.541252] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4326 to 0x280000400:4353) [13103.536451] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13105.347303] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [13114.387772] Lustre: DEBUG MARKER: == replay-single test 100d: DNE: cancel update logs upon recovery abort ========================================================== 07:11:50 (1743505910) [13121.518776] Lustre: Failing over lustre-MDT0000 [13127.271488] Lustre: server umount lustre-MDT0000 complete [13132.507714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13132.758650] Lustre: lustre-MDT0000: Aborting client recovery [13132.761361] LustreError: 271851:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [13132.761706] LustreError: 271881:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0000-osd: get update log duration 0, retries 0, failed: rc = -108 [13132.764899] Lustre: 271883:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [13132.767437] LustreError: 271881:0:(lod_dev.c:506:lod_sub_recovery_thread()) Skipped 1 previous similar message [13132.773861] Lustre: 271883:0:(ldlm_lib.c:2289:target_recovery_overseer()) Skipped 2 previous similar messages [13132.778997] Lustre: 271883:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [13132.783212] Lustre: lustre-MDT0000: disconnecting 2 stale clients [13132.787590] Lustre: lustre-MDT0000-osd: cancel update llog [0x20001a210:0x1:0x0] [13132.792199] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ec:0x1:0x0] [13132.817979] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9516 to 0x2c0000401:9537) [13132.818033] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9556 to 0x280000401:9577) [13134.960378] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13137.894072] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [13153.844140] Lustre: DEBUG MARKER: == replay-single test 100e: DNE: create striped dir on MDT0 and MDT1, fail MDT0, MDT1 ========================================================== 07:12:30 (1743505950) [13158.361874] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [13162.628510] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [13163.936649] Lustre: Failing over lustre-MDT0000 [13164.249309] Lustre: server umount lustre-MDT0000 complete [13166.391801] LustreError: 144817:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743505964 with bad export cookie 7018337741252296943 [13166.392971] Lustre: Failing over lustre-MDT0001 [13166.397086] LustreError: 144817:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [13166.642762] Lustre: server umount lustre-MDT0001 complete [13186.610795] LDISKFS-fs (dm-1): recovery complete [13186.613628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13186.790950] LustreError: 274766:0:(llog.c:1515:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [13186.796665] Lustre: 274766:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [13186.811990] LDISKFS-fs (dm-0): recovery complete [13186.813461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13191.136512] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9625d4f900 x1828185099763328/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [13195.087897] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13195.677251] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13195.742156] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9516 to 0x2c0000401:9569) [13195.744173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9556 to 0x280000401:9609) [13196.582445] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4326 to 0x280000400:4385) [13196.582879] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4198 to 0x2c0000400:4257) [13202.689222] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13204.820116] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13206.928351] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [13216.717907] Lustre: DEBUG MARKER: == replay-single test 101: Shouldn't reassign precreated objs to other files after recovery ========================================================== 07:13:32 (1743506012) [13221.600639] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [13230.416253] Lustre: Failing over lustre-MDT0000 [13230.834610] Lustre: server umount lustre-MDT0000 complete [13238.537164] LDISKFS-fs (dm-0): recovery complete [13238.538465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13238.744323] Lustre: lustre-MDT0000: Aborting client recovery [13238.745392] LustreError: 277059:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [13238.747086] Lustre: 277091:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [13238.748952] Lustre: 277091:0:(ldlm_lib.c:2289:target_recovery_overseer()) Skipped 2 previous similar messages [13238.750613] Lustre: 277091:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [13238.753384] Lustre: 277091:0:(genops.c:1508:class_disconnect_stale_exports()) Skipped 1 previous similar message [13238.755257] Lustre: lustre-MDT0000: disconnecting 2 stale clients [13238.759295] Lustre: lustre-MDT0000-osd: cancel update llog [0x200033080:0x1:0x0] [13238.766717] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x24000bb99:0x1:0x0] [13238.785790] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9611 to 0x280000401:10153) [13238.785953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9516 to 0x2c0000401:10113) [13241.404961] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13243.878181] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [13294.307488] Lustre: DEBUG MARKER: == replay-single test 102a: check resend (request lost) with multiple modify RPCs in flight ========================================================== 07:14:50 (1743506090) [13295.606543] Lustre: *** cfs_fail_loc=159, val=0*** [13295.608311] Lustre: Skipped 1 previous similar message [13311.757562] Lustre: lustre-MDT0000: Client 7d8d0c82-6141-4101-872b-23cb3da12043 (at 192.168.206.51@tcp) reconnecting [13311.760049] Lustre: Skipped 1 previous similar message [13320.877090] Lustre: DEBUG MARKER: == replay-single test 102b: check resend (reply lost) with multiple modify RPCs in flight ========================================================== 07:15:16 (1743506116) [13346.871877] Lustre: DEBUG MARKER: == replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight ========================================================== 07:15:42 (1743506142) [13352.555696] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [13356.067758] Lustre: Failing over lustre-MDT0000 [13356.384753] Lustre: server umount lustre-MDT0000 complete [13374.901102] LDISKFS-fs (dm-0): recovery complete [13374.902466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13377.872436] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13380.627934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10661 to 0x280000401:10697) [13380.628282] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10621 to 0x2c0000401:10657) [13384.470073] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13386.925887] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13398.456221] Lustre: DEBUG MARKER: == replay-single test 102d: check replay [13403.127537] Lustre: Failing over lustre-MDT0001 [13403.343563] Lustre: server umount lustre-MDT0001 complete [13419.223886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13419.292707] Lustre: 281007:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [13419.295249] Lustre: 281007:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 7 previous similar messages [13421.936748] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13424.664870] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4289) [13424.664938] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4417) [13427.982327] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13430.036925] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [13439.807816] Lustre: DEBUG MARKER: == replay-single test 103: Check otr_next_id overflow ==== 07:17:16 (1743506236) [13443.178462] Lustre: Failing over lustre-MDT0000 [13443.401606] Lustre: server umount lustre-MDT0000 complete [13459.364952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13462.144205] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13465.093272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:10729) [13465.093365] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:10689) [13468.947484] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13471.466159] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13482.816536] Lustre: DEBUG MARKER: == replay-single test 110a: DNE: create striped dir, fail MDT1 ========================================================== 07:17:58 (1743506278) [13487.725320] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [13488.978558] Lustre: Failing over lustre-MDT0000 [13489.299049] Lustre: server umount lustre-MDT0000 complete [13506.512092] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743506288/real 1743506288] req@ffff9e961ff49180 x1828185100464384/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1743506304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [13506.518457] Lustre: 3679:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 24 previous similar messages [13506.521988] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13506.525754] LustreError: Skipped 6 previous similar messages [13507.151545] LDISKFS-fs (dm-0): recovery complete [13507.153854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13516.769887] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9637119d00 x1828185100472960/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [13519.244428] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13521.935050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:10761) [13521.935080] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:10721) [13525.665921] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13527.826260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13537.712185] Lustre: DEBUG MARKER: == replay-single test 110b: DNE: create striped dir, fail MDT1 and client ========================================================== 07:18:53 (1743506333) [13543.055574] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [13544.677990] Lustre: Failing over lustre-MDT0000 [13544.893504] Lustre: server umount lustre-MDT0000 complete [13563.035325] LDISKFS-fs (dm-0): recovery complete [13563.037737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13565.742606] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13571.994397] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13574.731469] Lustre: lustre-MDT0000: Denying connection for new client 741d7d3a-4a18-43c5-b51e-bbd13e1bdcfd (at 192.168.206.51@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [13574.738145] Lustre: Skipped 5 previous similar messages [13580.046380] Lustre: lustre-MDT0000: Denying connection for new client 741d7d3a-4a18-43c5-b51e-bbd13e1bdcfd (at 192.168.206.51@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:57 [13580.055016] Lustre: Skipped 1 previous similar message [13590.286121] Lustre: lustre-MDT0000: Denying connection for new client 741d7d3a-4a18-43c5-b51e-bbd13e1bdcfd (at 192.168.206.51@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:47 [13590.293657] Lustre: Skipped 1 previous similar message [13610.765694] Lustre: lustre-MDT0000: Denying connection for new client 741d7d3a-4a18-43c5-b51e-bbd13e1bdcfd (at 192.168.206.51@tcp), waiting for 2 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 0:27 [13610.769885] Lustre: Skipped 3 previous similar messages [13638.000224] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [13638.003988] Lustre: 285929:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 7d8d0c82-6141-4101-872b-23cb3da12043@ [13638.010829] Lustre: 285929:0:(genops.c:1508:class_disconnect_stale_exports()) Skipped 1 previous similar message [13638.015630] Lustre: lustre-MDT0000: disconnecting 1 stale clients [13638.026207] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [13638.026371] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [13638.029577] Lustre: Skipped 8 previous similar messages [13638.031658] Lustre: Skipped 41 previous similar messages [13638.050768] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:10753) [13638.050786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:10793) [13651.386451] Lustre: DEBUG MARKER: == replay-single test 110c: DNE: create striped dir, fail MDT2 ========================================================== 07:20:47 (1743506447) [13656.864105] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [13658.184845] Lustre: Failing over lustre-MDT0001 [13658.293798] Lustre: server umount lustre-MDT0001 complete [13660.642771] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13660.644286] LustreError: 275592:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13660.645783] Lustre: Skipped 43 previous similar messages [13660.645939] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13660.653609] LustreError: 275592:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 153 previous similar messages [13660.655316] LustreError: Skipped 10 previous similar messages [13676.816870] LDISKFS-fs (dm-1): recovery complete [13676.818314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13677.005729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13677.010275] Lustre: Skipped 12 previous similar messages [13677.035611] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [13677.037559] Lustre: Skipped 16 previous similar messages [13677.326229] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [13677.329093] Lustre: Skipped 10 previous similar messages [13679.611357] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13682.182600] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4449) [13682.183274] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4321) [13685.916065] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13687.957986] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [13697.938099] Lustre: DEBUG MARKER: == replay-single test 110d: DNE: create striped dir, fail MDT2 and client ========================================================== 07:21:33 (1743506493) [13703.157301] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [13704.658408] Lustre: Failing over lustre-MDT0001 [13704.763831] Lustre: server umount lustre-MDT0001 complete [13723.418067] LDISKFS-fs (dm-1): recovery complete [13723.420367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13726.214095] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13732.745736] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13735.445823] Lustre: lustre-MDT0001: Denying connection for new client 3188be92-5ec8-41b8-bf29-feb5c1c20f75 (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [13735.456676] Lustre: Skipped 5 previous similar messages [13798.000173] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [13798.003259] Lustre: 289522:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 741d7d3a-4a18-43c5-b51e-bbd13e1bdcfd@ [13798.009160] Lustre: lustre-MDT0001: disconnecting 1 stale clients [13798.039694] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4353) [13798.039798] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4481) [13810.354789] Lustre: DEBUG MARKER: == replay-single test 110e: DNE: create striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 07:23:26 (1743506606) [13815.924427] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [13821.151192] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [13822.468524] Lustre: Failing over lustre-MDT0000 [13822.607535] Lustre: server umount lustre-MDT0000 complete [13824.960474] LustreError: 177653:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506623 with bad export cookie 7018337741252501700 [13824.962580] Lustre: Failing over lustre-MDT0001 [13824.964987] LustreError: 177653:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [13825.154074] Lustre: server umount lustre-MDT0001 complete [13845.754439] LDISKFS-fs (dm-1): recovery complete [13845.755800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13845.928052] LustreError: 292330:0:(llog.c:1515:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [13845.932851] Lustre: 292330:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [13846.004348] LDISKFS-fs (dm-0): recovery complete [13846.006684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13850.528594] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e963708b9c0 x1828185100661120/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [13852.986697] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:10785) [13852.998245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:10825) [13853.938793] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13854.345397] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13860.787512] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13863.346222] Lustre: lustre-MDT0001: Denying connection for new client add72148-909b-4fbf-8c6b-867cf3f88c2f (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [13863.350789] Lustre: Skipped 13 previous similar messages [13922.000181] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [13922.001854] Lustre: 292409:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 3188be92-5ec8-41b8-bf29-feb5c1c20f75@ [13922.004940] Lustre: lustre-MDT0001: disconnecting 1 stale clients [13922.028993] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4513) [13922.029116] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4385) [13933.917434] Lustre: DEBUG MARKER: SKIP: replay-single test_110f skipping excluded test 110f [13935.794766] Lustre: DEBUG MARKER: == replay-single test 110g: DNE: create striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 07:25:31 (1743506731) [13940.729831] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [13945.342839] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [13946.560551] Lustre: Failing over lustre-MDT0000 [13946.701205] Lustre: server umount lustre-MDT0000 complete [13948.998634] LustreError: 271313:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743506747 with bad export cookie 7018337741252506089 [13949.000036] Lustre: Failing over lustre-MDT0001 [13949.009888] LustreError: 271313:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [13949.183607] Lustre: server umount lustre-MDT0001 complete [13970.650637] LDISKFS-fs (dm-0): recovery complete [13970.653293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13970.743562] LDISKFS-fs (dm-1): recovery complete [13970.746293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13973.984700] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e951cf5bf80 x1828185100728832/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [13977.557332] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13977.636629] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4417) [13977.637598] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4545) [13978.024256] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [13984.937865] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13993.229978] Lustre: lustre-MDT0000: Denying connection for new client 59c05d61-a246-48dc-933d-c314df31b43e (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:52 [13993.234008] Lustre: Skipped 14 previous similar messages [14046.000226] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [14046.002025] Lustre: 295710:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client add72148-909b-4fbf-8c6b-867cf3f88c2f@ [14046.004764] Lustre: lustre-MDT0000: disconnecting 1 stale clients [14046.025686] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:10817) [14046.025887] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:10857) [14059.087240] Lustre: DEBUG MARKER: == replay-single test 111a: DNE: unlink striped dir, fail MDT1 ========================================================== 07:27:35 (1743506855) [14064.653452] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [14065.935470] Lustre: Failing over lustre-MDT0000 [14066.310404] Lustre: server umount lustre-MDT0000 complete [14084.781570] LDISKFS-fs (dm-0): recovery complete [14084.783429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14087.497380] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14090.255283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:10889) [14090.255386] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:10849) [14094.143623] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14096.498789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14108.155391] Lustre: DEBUG MARKER: == replay-single test 111b: DNE: unlink striped dir, fail MDT2 ========================================================== 07:28:24 (1743506904) [14113.626311] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [14115.304432] Lustre: Failing over lustre-MDT0001 [14115.416693] Lustre: server umount lustre-MDT0001 complete [14133.662537] LDISKFS-fs (dm-1): recovery complete [14133.664701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14133.717463] Lustre: 299649:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [14133.720119] Lustre: 299649:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 3 previous similar messages [14136.432476] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14142.671621] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14209.000102] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [14209.001620] Lustre: 299673:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client 59c05d61-a246-48dc-933d-c314df31b43e@ [14209.004874] Lustre: lustre-MDT0001: disconnecting 1 stale clients [14209.033465] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4577) [14209.033574] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4449) [14221.640839] Lustre: DEBUG MARKER: == replay-single test 111c: DNE: unlink striped dir, uncommit on MDT1, fail client/MDT1/MDT2 ========================================================== 07:30:17 (1743507017) [14227.217909] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [14232.459969] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [14233.627758] Lustre: Failing over lustre-MDT0000 [14233.757390] Lustre: server umount lustre-MDT0000 complete [14236.012340] LustreError: 271313:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507034 with bad export cookie 7018337741252510800 [14236.014100] Lustre: Failing over lustre-MDT0001 [14236.017858] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14236.017865] LustreError: Skipped 4 previous similar messages [14236.017934] LustreError: 271313:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [14236.641940] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14236.643943] Lustre: Skipped 7 previous similar messages [14241.926882] Lustre: server umount lustre-MDT0001 complete [14263.089585] LDISKFS-fs (dm-1): recovery complete [14263.091916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14263.242317] LustreError: 302480:0:(llog.c:1515:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [14263.244818] Lustre: 302480:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [14263.317699] LDISKFS-fs (dm-0): recovery complete [14263.319188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14266.336956] LustreError: 302500:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14266.340987] LustreError: 302500:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 62 previous similar messages [14281.696563] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e9625cfa2c0 x1828185100893440/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [14281.843499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14281.846743] Lustre: Skipped 7 previous similar messages [14281.867701] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [14281.869697] Lustre: Skipped 7 previous similar messages [14281.992789] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [14281.994823] LustreError: Skipped 5 previous similar messages [14282.012658] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 1 client reconnects [14282.014122] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [14282.014947] Lustre: Skipped 7 previous similar messages [14282.018092] Lustre: Skipped 25 previous similar messages [14285.451401] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14285.712983] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14288.063885] Lustre: lustre-MDT0001: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [14288.067775] Lustre: Skipped 8 previous similar messages [14288.082418] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4481) [14288.082538] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4609) [14292.783324] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14295.630590] Lustre: lustre-MDT0000: Denying connection for new client fab0233e-69e1-4b8f-9d9d-21f8342d9bf6 (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [14295.634674] Lustre: Skipped 24 previous similar messages [14357.000204] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [14357.002627] Lustre: 302600:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client d0dd9c56-99de-49b2-a2fb-2c596ea7b0ef@ [14357.005623] Lustre: lustre-MDT0000: disconnecting 1 stale clients [14357.028274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:10921) [14357.028616] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:10881) [14367.767834] Lustre: DEBUG MARKER: == replay-single test 111d: DNE: unlink striped dir, uncommit on MDT2, fail client/MDT1/MDT2 ========================================================== 07:32:43 (1743507163) [14373.087754] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [14378.367214] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [14379.630166] Lustre: Failing over lustre-MDT0000 [14379.752112] Lustre: server umount lustre-MDT0000 complete [14380.519331] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14380.522744] Lustre: Skipped 24 previous similar messages [14382.277993] LustreError: 144817:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507180 with bad export cookie 7018337741252513698 [14382.278675] Lustre: Failing over lustre-MDT0001 [14382.283931] LustreError: 144817:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message [14382.468690] Lustre: server umount lustre-MDT0001 complete [14400.993169] Lustre: 3681:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743507183/real 1743507183] req@ffff9e951c6b6d80 x1828185100960256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743507199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [14401.005449] Lustre: 3681:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 42 previous similar messages [14405.075404] LDISKFS-fs (dm-1): recovery complete [14405.077573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14405.080281] LDISKFS-fs (dm-0): recovery complete [14405.081992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14405.253911] LustreError: 305762:0:(llog.c:1515:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [14405.258231] Lustre: 305762:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [14407.136771] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e951c6b4b00 x1828185100963072/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [14410.523209] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14410.933876] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14411.770833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:10953) [14411.771508] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:10913) [14418.212428] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14481.000230] Lustre: lustre-MDT0001: recovery is timed out, evict stale exports [14481.042691] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4641) [14481.042805] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4513) [14491.398847] Lustre: DEBUG MARKER: == replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 ========================================================== 07:34:47 (1743507287) [14496.853448] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [14501.644560] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [14502.970423] Lustre: Failing over lustre-MDT0000 [14503.156205] Lustre: server umount lustre-MDT0000 complete [14505.470254] LustreError: 42091:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507303 with bad export cookie 7018337741252515882 [14505.471750] Lustre: Failing over lustre-MDT0001 [14505.473501] LustreError: 42091:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 7 previous similar messages [14505.637195] Lustre: server umount lustre-MDT0001 complete [14527.007505] LDISKFS-fs (dm-1): recovery complete [14527.009084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14527.141197] LDISKFS-fs (dm-0): recovery complete [14527.143507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14527.171747] LustreError: 308992:0:(llog.c:1515:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [14527.175795] Lustre: 308992:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [14530.016455] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e951362f900 x1828185101028736/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [14533.706294] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14534.131780] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14535.403427] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4673) [14535.403444] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4545) [14542.580099] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:10945) [14542.580140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:10985) [14546.659517] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14549.000592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14551.048784] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [14562.141662] Lustre: DEBUG MARKER: == replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 ========================================================== 07:35:57 (1743507357) [14567.394280] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [14572.225891] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [14573.442986] Lustre: Failing over lustre-MDT0000 [14573.537191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14573.543651] Lustre: Skipped 2 previous similar messages [14573.703500] Lustre: server umount lustre-MDT0000 complete [14576.218050] LustreError: 271313:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507374 with bad export cookie 7018337741252518171 [14576.219561] Lustre: Failing over lustre-MDT0001 [14576.222598] LustreError: 271313:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [14576.403711] Lustre: server umount lustre-MDT0001 complete [14598.232379] LDISKFS-fs (dm-0): recovery complete [14598.234684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14598.343871] LDISKFS-fs (dm-1): recovery complete [14598.346333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14601.184551] LustreError: 3678:0:(client.c:1292:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9e95141d6d80 x1828185101078528/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [14604.157238] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:10977) [14604.157246] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11017) [14604.250538] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4705) [14604.251243] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4577) [14604.905939] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14605.294028] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14612.086376] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14614.503753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14616.717189] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [14627.648707] Lustre: DEBUG MARKER: == replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 ========================================================== 07:37:03 (1743507423) [14632.651215] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [14637.155813] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [14638.512315] Lustre: Failing over lustre-MDT0000 [14638.823249] Lustre: server umount lustre-MDT0000 complete [14640.956119] LustreError: 271313:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743507439 with bad export cookie 7018337741252520446 [14640.957076] Lustre: Failing over lustre-MDT0001 [14640.959888] LustreError: 271313:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 4 previous similar messages [14641.119339] Lustre: server umount lustre-MDT0001 complete [14661.706386] LDISKFS-fs (dm-1): recovery complete [14661.708245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14661.885966] LustreError: 315624:0:(llog.c:1515:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [14661.890458] Lustre: 315624:0:(mgc_request_server.c:558:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [14661.933299] LDISKFS-fs (dm-0): recovery complete [14661.935423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14666.145478] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdad12f343 [14666.147631] Lustre: Skipped 1 previous similar message [14669.526097] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14669.897381] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14671.533288] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4609) [14671.534794] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4737) [14679.754443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11049) [14679.754447] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:11009) [14683.339869] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14685.192648] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14687.001965] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [14696.918674] Lustre: DEBUG MARKER: == replay-single test 112a: DNE: cross MDT rename, fail MDT1 ========================================================== 07:38:13 (1743507493) [14698.584732] Lustre: DEBUG MARKER: SKIP: replay-single test_112a needs >= 4 MDTs [14700.480840] Lustre: DEBUG MARKER: == replay-single test 112b: DNE: cross MDT rename, fail MDT2 ========================================================== 07:38:16 (1743507496) [14702.136801] Lustre: DEBUG MARKER: SKIP: replay-single test_112b needs >= 4 MDTs [14704.155495] Lustre: DEBUG MARKER: == replay-single test 112c: DNE: cross MDT rename, fail MDT3 ========================================================== 07:38:20 (1743507500) [14705.816070] Lustre: DEBUG MARKER: SKIP: replay-single test_112c needs >= 4 MDTs [14707.578878] Lustre: DEBUG MARKER: == replay-single test 112d: DNE: cross MDT rename, fail MDT4 ========================================================== 07:38:23 (1743507503) [14709.329552] Lustre: DEBUG MARKER: SKIP: replay-single test_112d needs >= 4 MDTs [14711.553915] Lustre: DEBUG MARKER: == replay-single test 112e: DNE: cross MDT rename, fail MDT1 and MDT2 ========================================================== 07:38:27 (1743507507) [14713.555281] Lustre: DEBUG MARKER: SKIP: replay-single test_112e needs >= 4 MDTs [14715.843940] Lustre: DEBUG MARKER: == replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 ========================================================== 07:38:31 (1743507511) [14717.671508] Lustre: DEBUG MARKER: SKIP: replay-single test_112f needs >= 4 MDTs [14719.993027] Lustre: DEBUG MARKER: == replay-single test 112g: DNE: cross MDT rename, fail MDT1 and MDT4 ========================================================== 07:38:35 (1743507515) [14722.308644] Lustre: DEBUG MARKER: SKIP: replay-single test_112g needs >= 4 MDTs [14724.381852] Lustre: DEBUG MARKER: == replay-single test 112h: DNE: cross MDT rename, fail MDT2 and MDT3 ========================================================== 07:38:40 (1743507520) [14726.698066] Lustre: DEBUG MARKER: SKIP: replay-single test_112h needs >= 4 MDTs [14728.875680] Lustre: DEBUG MARKER: == replay-single test 112i: DNE: cross MDT rename, fail MDT2 and MDT4 ========================================================== 07:38:45 (1743507525) [14731.117190] Lustre: DEBUG MARKER: SKIP: replay-single test_112i needs >= 4 MDTs [14733.360066] Lustre: DEBUG MARKER: == replay-single test 112j: DNE: cross MDT rename, fail MDT3 and MDT4 ========================================================== 07:38:49 (1743507529) [14735.567295] Lustre: DEBUG MARKER: SKIP: replay-single test_112j needs >= 4 MDTs [14737.947717] Lustre: DEBUG MARKER: == replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 ========================================================== 07:38:53 (1743507533) [14740.162680] Lustre: DEBUG MARKER: SKIP: replay-single test_112k needs >= 4 MDTs [14742.535448] Lustre: DEBUG MARKER: == replay-single test 112l: DNE: cross MDT rename, fail MDT1,MDT2,MDT4 ========================================================== 07:38:58 (1743507538) [14745.001174] Lustre: DEBUG MARKER: SKIP: replay-single test_112l needs >= 4 MDTs [14747.359385] Lustre: DEBUG MARKER: == replay-single test 112m: DNE: cross MDT rename, fail MDT1,MDT3,MDT4 ========================================================== 07:39:03 (1743507543) [14749.144297] Lustre: DEBUG MARKER: SKIP: replay-single test_112m needs >= 4 MDTs [14751.226596] Lustre: DEBUG MARKER: == replay-single test 112n: DNE: cross MDT rename, fail MDT2,MDT3,MDT4 ========================================================== 07:39:07 (1743507547) [14753.152080] Lustre: DEBUG MARKER: SKIP: replay-single test_112n needs >= 4 MDTs [14755.551559] Lustre: DEBUG MARKER: == replay-single test 115: failover for create/unlink striped directory ========================================================== 07:39:11 (1743507551) [14760.745482] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [14762.560310] Lustre: Failing over lustre-MDT0001 [14762.814793] Lustre: server umount lustre-MDT0001 complete [14780.517587] LDISKFS-fs (dm-1): recovery complete [14780.519672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14780.583298] Lustre: 319241:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [14780.588316] Lustre: 319241:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [14783.140971] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14786.093771] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4641) [14786.094314] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4769) [14789.823689] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14792.417090] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [14799.715689] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [14801.397840] Lustre: Failing over lustre-MDT0000 [14801.624640] Lustre: server umount lustre-MDT0000 complete [14818.982024] LDISKFS-fs (dm-0): recovery complete [14818.983351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14819.093291] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (not set up) [14819.095644] Lustre: Skipped 4 previous similar messages [14821.323799] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14824.475203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:11041) [14824.475416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11081) [14827.386330] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14829.166976] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14838.852996] Lustre: DEBUG MARKER: == replay-single test 116a: large update log master MDT recovery ========================================================== 07:40:35 (1743507635) [14843.430838] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [14843.977193] Lustre: *** cfs_fail_loc=1702, val=0*** [14845.319430] Lustre: Failing over lustre-MDT0000 [14845.536925] Lustre: server umount lustre-MDT0000 complete [14863.610227] LDISKFS-fs (dm-0): recovery complete [14863.611603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14863.678745] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14863.681400] LustreError: Skipped 5 previous similar messages [14866.169208] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14869.072630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11113) [14869.072690] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:11073) [14871.996568] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14873.780438] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14884.228734] Lustre: DEBUG MARKER: == replay-single test 116b: large update log slave MDT recovery ========================================================== 07:41:20 (1743507680) [14889.379900] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [14889.923691] Lustre: *** cfs_fail_loc=1702, val=0*** [14891.082660] Lustre: Failing over lustre-MDT0001 [14891.269748] Lustre: server umount lustre-MDT0001 complete [14892.816055] LustreError: 315648:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14892.820226] LustreError: 315648:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 74 previous similar messages [14894.562326] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14894.564286] LustreError: Skipped 9 previous similar messages [14908.732959] LDISKFS-fs (dm-1): recovery complete [14908.734435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [14908.915807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14908.917978] Lustre: Skipped 12 previous similar messages [14908.944799] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [14908.946604] Lustre: Skipped 12 previous similar messages [14910.122462] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [14910.125296] Lustre: Skipped 12 previous similar messages [14911.130880] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14914.023246] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [14914.026769] Lustre: Skipped 41 previous similar messages [14914.043877] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [14914.048366] Lustre: Skipped 12 previous similar messages [14914.066676] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4393 to 0x280000400:4801) [14914.067460] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4265 to 0x2c0000400:4673) [14917.084665] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14918.911023] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [14930.354321] Lustre: DEBUG MARKER: == replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2 ========================================================== 07:42:06 (1743507726) [14932.450417] Lustre: DEBUG MARKER: SKIP: replay-single test_117 needs >= 4 MDTs [14934.938190] Lustre: DEBUG MARKER: == replay-single test 118: invalidate osp update will not cause update log corruption ========================================================== 07:42:10 (1743507730) [14936.210805] Lustre: *** cfs_fail_loc=1705, val=0*** [14941.503352] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [14942.852894] Lustre: Failing over lustre-MDT0000 [14943.190316] Lustre: server umount lustre-MDT0000 complete [14960.936411] LDISKFS-fs (dm-0): recovery complete [14960.937721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14970.339945] Lustre: Evicted from MGS (at 192.168.206.151@tcp) after server handle changed from 0x0 to 0x616624fdad131b8a [14972.729046] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [14975.535106] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:11105) [14975.535110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11145) [14978.625498] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14981.011808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14989.669223] Lustre: DEBUG MARKER: == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 07:43:05 (1743507785) [14994.820438] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [14996.454549] Lustre: Failing over lustre-MDT0000 [14996.657095] Lustre: server umount lustre-MDT0000 complete [15001.065344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15001.070785] Lustre: Skipped 38 previous similar messages [15004.937347] LDISKFS-fs (dm-0): recovery complete [15004.938852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15007.310075] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15007.501756] Lustre: 315646:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [15010.275093] Lustre: 175905:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 [15010.282040] LustreError: 328594:0:(ldlm_lib.c:2596:replay_request_or_update()) cfs_fail_timeout id 714 sleeping for 65000ms [15011.551394] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15075.288104] LustreError: 328594:0:(ldlm_lib.c:2596:replay_request_or_update()) cfs_fail_timeout id 714 awake [15075.290167] Lustre: 328594:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c4f86513-5ca7-4012-9fdf-746d8e48d78f@192.168.206.51@tcp [15075.293242] Lustre: 328594:0:(genops.c:1508:class_disconnect_stale_exports()) Skipped 1 previous similar message [15075.295373] Lustre: lustre-MDT0000: disconnecting 1 stale clients [15075.296486] Lustre: Skipped 1 previous similar message [15075.298109] Lustre: 328594:0:(ldlm_lib.c:1801:abort_req_replay_queue()) @@@ aborted: req@ffff9e951d347340 x1828185096534912/t0(519691042821) o36->c4f86513-5ca7-4012-9fdf-746d8e48d78f@192.168.206.51@tcp:731/0 lens 528/0 e 7 to 0 dl 1743507886 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'mcreate.0' uid:0 gid:0 [15075.303659] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [15075.305507] Lustre: 328594:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 [15075.309709] Lustre: lustre-MDT0000: Denying connection for new client c4f86513-5ca7-4012-9fdf-746d8e48d78f (at 192.168.206.51@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 1 evicted) already passed deadline 0:08 [15075.318884] Lustre: Skipped 25 previous similar messages [15075.344314] Lustre: 328594:0:(ldlm_lib.c:2279:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [15075.346750] Lustre: 328594:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [15075.348818] Lustre: 328594:0:(ldlm_lib.c:2289:target_recovery_overseer()) Skipped 2 previous similar messages [15075.358348] Lustre: lustre-MDT0000-osd: cancel update llog [0x200034020:0x1:0x0] [15075.363694] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x24000c36a:0x1:0x0] [15075.378879] Lustre: 328594:0:(ldlm_lib.c:2854:target_recovery_thread()) too long recovery - read logs [15075.382127] LustreError: dumping log to /tmp/lustre-log.1743507873.328594 [15075.463379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11177) [15075.463511] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:11137) [15082.156485] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 62 sec [15093.108767] Lustre: DEBUG MARKER: == replay-single test 120: DNE fail abort should stop both normal and DNE replay ========================================================== 07:44:48 (1743507888) [15097.373051] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [15100.552397] Lustre: Failing over lustre-MDT0000 [15106.144860] Lustre: server umount lustre-MDT0000 complete [15113.740297] LDISKFS-fs (dm-0): recovery complete [15113.742462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15113.973588] Lustre: lustre-MDT0000: Aborting client recovery [15113.974960] LustreError: 330436:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [15113.975977] LustreError: 330467:0:(lod_dev.c:506:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -108 [15113.977251] Lustre: 330468:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [15113.984670] Lustre: 330468:0:(ldlm_lib.c:2289:target_recovery_overseer()) Skipped 1 previous similar message [15113.991341] Lustre: lustre-MDT0000-osd: cancel update llog [0x20003bd20:0x3:0x0] [15113.999763] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400128fa:0x1:0x0] [15114.030212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11209) [15114.030301] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10673 to 0x2c0000401:11169) [15116.551712] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15119.340718] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [15136.850619] Lustre: DEBUG MARKER: == replay-single test 121: lock replay timed out and race ========================================================== 07:45:32 (1743507932) [15139.264316] Lustre: Failing over lustre-MDT0000 [15139.400367] Lustre: server umount lustre-MDT0000 complete [15144.416552] Lustre: *** cfs_fail_loc=721, val=0*** [15144.417714] Lustre: Skipped 6 previous similar messages [15144.930372] Lustre: *** cfs_fail_loc=721, val=0*** [15144.931763] Lustre: Skipped 3 previous similar messages [15146.824333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15146.906447] Lustre: *** cfs_fail_loc=721, val=0*** [15146.908384] Lustre: Skipped 4 previous similar messages [15149.365404] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15149.536452] Lustre: *** cfs_fail_loc=721, val=0*** [15149.537639] Lustre: Skipped 86 previous similar messages [15152.108204] Lustre: *** cfs_fail_loc=721, val=1*** [15152.109355] Lustre: Skipped 22 previous similar messages [15154.656662] Lustre: *** cfs_fail_loc=721, val=1*** [15154.658560] Lustre: Skipped 33 previous similar messages [15164.896669] Lustre: *** cfs_fail_loc=721, val=1*** [15164.898583] Lustre: Skipped 34 previous similar messages [15167.256984] Lustre: lustre-MDT0000: Client c4f86513-5ca7-4012-9fdf-746d8e48d78f (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:52 [15182.304109] Lustre: 3678:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743507950/real 1743507950] req@ffff9e961fd38040 x1828185101549824/t0(0) o400->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 224/224 e 0 to 1 dl 1743507980 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 [15182.309952] Lustre: 3678:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 56 previous similar messages [15182.314423] Lustre: *** cfs_fail_loc=721, val=1*** [15182.315401] Lustre: Skipped 57 previous similar messages [15182.316363] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [15182.607559] Lustre: lustre-MDT0000: Client c4f86513-5ca7-4012-9fdf-746d8e48d78f (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [15198.990963] Lustre: lustre-MDT0000: Client c4f86513-5ca7-4012-9fdf-746d8e48d78f (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:21 [15212.512644] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [15214.349337] Lustre: *** cfs_fail_loc=721, val=1*** [15214.350877] Lustre: Skipped 134 previous similar messages [15215.375710] Lustre: lustre-MDT0000: Client c4f86513-5ca7-4012-9fdf-746d8e48d78f (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:17 [15242.720655] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [15242.723674] Lustre: *** cfs_fail_loc=721, val=1*** [15242.725236] Lustre: Skipped 2 previous similar messages [15242.726539] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [15262.477785] Lustre: lustre-MDT0000: Client c4f86513-5ca7-4012-9fdf-746d8e48d78f (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:10 [15262.483403] Lustre: Skipped 2 previous similar messages [15272.928971] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [15278.868358] Lustre: *** cfs_fail_loc=721, val=1*** [15278.870513] Lustre: Skipped 246 previous similar messages [15295.245673] Lustre: lustre-MDT0000: Recovery already passed deadline 0:02. If you do not want to wait more, you may force taget eviction via 'lctl --device lustre-MDT0000 abort_recovery. [15303.136975] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [15303.139813] Lustre: 331866:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [15303.142439] Lustre: 331866:0:(ldlm_lib.c:1971:extend_recovery_timer()) Skipped 20 previous similar messages [15326.995342] Lustre: lustre-MDT0000: Client c4f86513-5ca7-4012-9fdf-746d8e48d78f (at 192.168.206.51@tcp) reconnected, waiting for 2 clients in recovery for 0:03 [15327.000520] Lustre: Skipped 2 previous similar messages [15333.344818] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [15333.348346] Lustre: 331866:0:(ldlm_lib.c:1971:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [15333.351210] Lustre: 331866:0:(ldlm_lib.c:2279:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [15333.353383] Lustre: 331866:0:(ldlm_lib.c:2279:target_recovery_overseer()) Skipped 1 previous similar message [15333.355417] Lustre: 331866:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [15333.357795] Lustre: 331866:0:(ldlm_lib.c:2289:target_recovery_overseer()) Skipped 2 previous similar messages [15333.359909] Lustre: 331866:0:(genops.c:1508:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client c4f86513-5ca7-4012-9fdf-746d8e48d78f@192.168.206.51@tcp [15333.362773] Lustre: 331866:0:(genops.c:1508:class_disconnect_stale_exports()) Skipped 2 previous similar messages [15333.364976] Lustre: lustre-MDT0000: disconnecting 1 stale clients [15333.366261] Lustre: Skipped 1 previous similar message [15333.367709] LustreError: 331866:0:(ldlm_lib.c:1821:abort_lock_replay_queue()) @@@ aborted: req@ffff9e9636e08bc0 x1828185096681344/t0(0) o101->c4f86513-5ca7-4012-9fdf-746d8e48d78f@192.168.206.51@tcp:0/0 lens 328/0 e 0 to 0 dl 1743508008 ref 1 fl Complete:/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 [15333.381034] Lustre: lustre-MDT0000-osd: cancel update llog [0x20003c4f0:0x1:0x0] [15333.388883] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400128fb:0x1:0x0] [15333.399859] Lustre: 331866:0:(ldlm_lib.c:2854:target_recovery_thread()) too long recovery - read logs [15333.402072] LustreError: dumping log to /tmp/lustre-log.1743508131.331866 [15333.449815] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11171 to 0x2c0000401:11201) [15333.449823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11241) [15350.845767] Lustre: DEBUG MARKER: == replay-single test 130a: DoM file create (setstripe) replay ========================================================== 07:49:06 (1743508146) [15355.996342] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [15357.202193] Lustre: Failing over lustre-MDT0000 [15357.315468] Lustre: server umount lustre-MDT0000 complete [15374.896032] LDISKFS-fs (dm-0): recovery complete [15374.897351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15377.485243] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15380.519856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11273) [15380.519875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11171 to 0x2c0000401:11233) [15383.549876] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15385.362907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15394.280520] Lustre: DEBUG MARKER: == replay-single test 130b: DoM file create (inherited) replay ========================================================== 07:49:50 (1743508190) [15399.175704] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [15400.395992] Lustre: Failing over lustre-MDT0000 [15400.597293] Lustre: server umount lustre-MDT0000 complete [15418.199328] LDISKFS-fs (dm-0): recovery complete [15418.200574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15420.824910] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15423.507877] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11171 to 0x2c0000401:11265) [15423.507881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11305) [15426.446542] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15428.192489] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15437.202363] Lustre: DEBUG MARKER: == replay-single test 131a: DoM file write lock replay === 07:50:33 (1743508233) [15441.932775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [15443.051347] Lustre: Failing over lustre-MDT0000 [15443.219071] Lustre: server umount lustre-MDT0000 complete [15460.150743] LDISKFS-fs (dm-0): recovery complete [15460.152062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15462.475297] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15465.484463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10713 to 0x280000401:11337) [15465.485108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11171 to 0x2c0000401:11297) [15468.220450] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15469.989498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15479.308696] Lustre: DEBUG MARKER: SKIP: replay-single test_131b skipping excluded test 131b [15481.275788] Lustre: DEBUG MARKER: == replay-single test 132a: PFL new component instantiate replay ========================================================== 07:51:17 (1743508277) [15485.740937] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [15487.014723] Lustre: Failing over lustre-MDT0000 [15487.223279] Lustre: server umount lustre-MDT0000 complete [15496.161155] LustreError: 315648:0:(ldlm_lib.c:1095:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15496.168417] LustreError: 315648:0:(ldlm_lib.c:1095:target_handle_connect()) Skipped 133 previous similar messages [15504.557962] LDISKFS-fs (dm-0): recovery complete [15504.559251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15504.633490] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15504.636510] LustreError: Skipped 7 previous similar messages [15507.009699] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15510.026390] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11329) [15510.026540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11340 to 0x280000401:11369) [15512.902699] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15514.741832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15523.900582] Lustre: DEBUG MARKER: == replay-single test 133: check resend of ongoing requests for lwp during failover ========================================================== 07:51:59 (1743508319) [15528.469644] Lustre: *** cfs_fail_loc=123, val=2147483648*** [15528.471052] Lustre: Skipped 265 previous similar messages [15530.557544] Lustre: Failing over lustre-MDT0000 [15530.787681] Lustre: server umount lustre-MDT0000 complete [15535.587026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15535.590614] LustreError: Skipped 7 previous similar messages [15545.101713] Lustre: lustre-MDT0001: Client 6dc09a86-3267-4417-a1b9-8fa2066376dc (at 192.168.206.51@tcp) reconnecting [15545.104431] Lustre: Skipped 1 previous similar message [15545.820623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15546.014691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15546.016597] Lustre: Skipped 8 previous similar messages [15546.051533] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [15546.053453] Lustre: Skipped 10 previous similar messages [15547.149722] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [15547.153057] Lustre: Skipped 7 previous similar messages [15548.179582] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15551.461224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.206.151@tcp (at 0@lo) [15551.464102] Lustre: Skipped 35 previous similar messages [15551.467967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:1:mdt [15551.471566] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:1:mdt] [15551.475928] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [15551.479191] Lustre: Skipped 7 previous similar messages [15551.495785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11340 to 0x280000401:11401) [15551.495872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11361) [15560.419824] Lustre: DEBUG MARKER: == replay-single test 134: replay creation of a file created in a pool ========================================================== 07:52:36 (1743508356) [15571.326394] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [15572.577586] Lustre: Failing over lustre-MDT0000 [15572.755198] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [15572.757139] Lustre: Skipped 6 previous similar messages [15572.927882] Lustre: server umount lustre-MDT0000 complete [15590.597448] LDISKFS-fs (dm-0): recovery complete [15590.599622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15593.195427] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15596.039640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11340 to 0x280000401:11433) [15596.039654] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11393) [15598.918635] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15600.644521] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15616.613750] Lustre: DEBUG MARKER: == replay-single test 135: Server failure in lock replay phase ========================================================== 07:53:32 (1743508412) [15618.020629] Lustre: Failing over lustre-OST0000 [15618.067212] Lustre: server umount lustre-OST0000 complete [15619.555058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15619.561349] Lustre: Skipped 34 previous similar messages [15633.064706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15633.110508] Lustre: 343898:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.206.151@tcp: no remote llog for lustre-sptlrpc, check MGS config [15633.113286] Lustre: 343898:0:(mgc_request_server.c:553:mgc_llog_local_copy()) Skipped 1 previous similar message [15636.562153] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15642.068926] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15643.801693] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [15651.797582] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [15653.242825] Lustre: Failing over lustre-OST0000 [15653.294141] Lustre: server umount lustre-OST0000 complete [15656.734553] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [15662.714448] LDISKFS-fs (dm-2): recovery complete [15662.716340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15664.279422] Lustre: *** cfs_fail_loc=32d, val=20*** [15664.281031] Lustre: Skipped 3 previous similar messages [15665.918129] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15670.631498] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount REPLAY_LOCKS osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15672.353461] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in REPLAY_LOCKS state after 0 sec [15673.752654] Lustre: Failing over lustre-OST0000 [15673.755663] LustreError: 346577:0:(ldlm_lib.c:2907:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery [15673.757928] Lustre: 345911:0:(ldlm_lib.c:2289:target_recovery_overseer()) recovery is aborted, evict exports in recovery [15673.760441] LustreError: 345911:0:(ofd_obd.c:1298:ofd_iocontrol()) lustre-OST0000: iocontrol from 'tgt_recover_0' cmd=c00866c1 _IOWR('f', 193, 8) unrecognized: rc = -25 [15673.804501] Lustre: server umount lustre-OST0000 complete [15687.722478] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [15691.255604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15694.632936] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15702.787247] Lustre: server umount lustre-OST0000 complete [15719.392085] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15719.533018] Lustre: server umount lustre-OST0001 complete [15724.411121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15727.835399] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15732.581367] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15735.847102] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15747.046406] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [15747.050856] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [15747.055983] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [15747.060522] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [15755.407879] Lustre: DEBUG MARKER: == replay-single test 136: MDS to disconnect all OSPs first, then cleanup ldlm ========================================================== 07:55:51 (1743508551) [15756.990939] Lustre: DEBUG MARKER: SKIP: replay-single test_136 needs > 2 MDTs [15758.700890] Lustre: DEBUG MARKER: == replay-single test 137a: DNE: create under striped dir, fail MDT1 ========================================================== 07:55:54 (1743508554) [15763.185450] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [15764.294492] Lustre: Failing over lustre-MDT0000 [15764.610394] Lustre: server umount lustre-MDT0000 complete [15781.591055] LDISKFS-fs (dm-0): recovery complete [15781.592410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15783.960935] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15787.018184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11455 to 0x280000401:11497) [15787.019105] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11425) [15789.713290] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15791.412359] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15800.089771] Lustre: DEBUG MARKER: == replay-single test 137b: DNE: create under striped dir, fail MDT2 ========================================================== 07:56:36 (1743508596) [15804.730631] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [15805.829098] Lustre: Failing over lustre-MDT0001 [15806.041601] Lustre: server umount lustre-MDT0001 complete [15823.109736] LDISKFS-fs (dm-1): recovery complete [15823.110959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15825.320678] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15828.489509] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4803 to 0x280000400:4833) [15828.489508] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4676 to 0x2c0000400:4705) [15831.312149] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15832.996804] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [15841.848374] Lustre: DEBUG MARKER: == replay-single test 137c: DNE: create under striped dir, fail MDT1/MDT2 ========================================================== 07:57:18 (1743508638) [15846.599611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [15850.714490] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [15851.956380] Lustre: Failing over lustre-MDT0001 [15852.170220] Lustre: server umount lustre-MDT0001 complete [15854.398183] Lustre: Failing over lustre-MDT0000 [15854.680397] Lustre: server umount lustre-MDT0000 complete [15875.172785] LDISKFS-fs (dm-1): recovery complete [15875.174289] LDISKFS-fs (dm-0): recovery complete [15875.174308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15875.176082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15876.448155] Lustre: 3681:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1743508657/real 1743508657] req@ffff9e951d203f80 x1828185101999104/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1743508673 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 [15876.460139] Lustre: 3681:0:(client.c:2346:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [15878.465255] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15878.774436] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15880.506434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11455 to 0x280000401:11529) [15880.507150] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11299 to 0x2c0000401:11457) [15880.553367] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4676 to 0x2c0000400:4737) [15880.555852] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4803 to 0x280000400:4865) [15884.720706] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15886.475371] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15888.077207] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [15896.898724] Lustre: DEBUG MARKER: == replay-single test 200: Dropping one OBD_PING should not cause disconnect ========================================================== 07:58:13 (1743508693) [15898.518584] Lustre: DEBUG MARKER: SKIP: replay-single test_200 Need remote client [15900.521239] Lustre: DEBUG MARKER: == replay-single test 201: MDT umount cascading disconnects timeouts ========================================================== 07:58:16 (1743508696) [15904.496160] LustreError: 357206:0:(tgt_handler.c:1099:tgt_disconnect()) cfs_fail_timeout id 245 sleeping for 8000ms [15911.182250] LustreError: 349879:0:(tgt_handler.c:1099:tgt_disconnect()) cfs_fail_timeout id 245 sleeping for 8000ms [15911.184378] LustreError: 349879:0:(tgt_handler.c:1099:tgt_disconnect()) Skipped 1 previous similar message [15912.504115] LustreError: 357206:0:(tgt_handler.c:1099:tgt_disconnect()) cfs_fail_timeout id 245 awake [15912.507737] Lustre: Failing over lustre-MDT0001 [15912.511515] LustreError: 348474:0:(tgt_handler.c:1099:tgt_disconnect()) cfs_fail_timeout id 245 sleeping for 8000ms [15912.513706] LustreError: 348474:0:(tgt_handler.c:1099:tgt_disconnect()) Skipped 2 previous similar messages [15919.192085] LustreError: 349707:0:(tgt_handler.c:1099:tgt_disconnect()) cfs_fail_timeout id 245 awake [15919.194594] LustreError: 349707:0:(tgt_handler.c:1099:tgt_disconnect()) Skipped 1 previous similar message [15920.520075] LustreError: 348465:0:(tgt_handler.c:1099:tgt_disconnect()) cfs_fail_timeout id 245 awake [15920.522172] LustreError: 348465:0:(tgt_handler.c:1099:tgt_disconnect()) Skipped 1 previous similar message [15920.563643] Lustre: server umount lustre-MDT0001 complete [15925.260572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [15927.485362] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15930.879165] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4676 to 0x2c0000400:4769) [15930.879225] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4803 to 0x280000400:4897) [15935.753810] Lustre: DEBUG MARKER: == replay-single test 202: pfl replay should recovery layout generation ========================================================== 07:58:51 (1743508731) [15941.358114] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [15942.420965] Lustre: Failing over lustre-MDT0000 [15942.636788] Lustre: server umount lustre-MDT0000 complete [15959.418349] LDISKFS-fs (dm-0): recovery complete [15959.419662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15961.619250] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [15964.689722] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11459 to 0x2c0000401:11489) [15964.689822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11455 to 0x280000401:11561) [15967.457501] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15969.173682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15977.401422] Lustre: DEBUG MARKER: == replay-single test complete, duration 15267 sec ======= 07:59:33 (1743508773) [15979.075493] Lustre: DEBUG MARKER: === replay-single: start cleanup 07:59:35 (1743508775) === [15987.826812] Lustre: DEBUG MARKER: === replay-single: finish cleanup 07:59:43 (1743508783) === [16021.550588] Lustre: server umount lustre-MDT0000 complete [16025.946087] LustreError: 177653:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1743508823 with bad export cookie 7018337741252570132 [16025.949913] LustreError: 177653:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 3 previous similar messages [16026.109795] Lustre: server umount lustre-MDT0001 complete [16040.963418] Lustre: server umount lustre-OST0000 complete [16045.287671] Lustre: server umount lustre-OST0001 complete [16060.706820] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [16063.355652] Key type lgssc unregistered [16063.674860] LNet: 362552:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16064.742566] LNet: Removed LNI 192.168.206.151@tcp [16065.711842] Key type .llcrypt unregistered [16065.713254] Key type ._llcrypt unregistered