[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 571832161 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002442] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007068] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009136] LSM: Security Framework initializing [ 0.010064] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.012055] *** VALIDATE selinux *** [ 0.020703] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025277] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027033] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029120] *** VALIDATE tmpfs *** [ 0.031433] *** VALIDATE proc *** [ 0.032246] *** VALIDATE cgroup *** [ 0.034005] *** VALIDATE cgroup2 *** [ 0.035263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036123] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038019] Spectre V2 : User space: Vulnerable [ 0.040005] Speculative Store Bypass: Vulnerable [ 0.043448] debug: unmapping init [mem 0xffffffff93659000-0xffffffff93660fff] [ 0.046152] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047658] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048043] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054007] ... event mask: 000000070000000f [ 0.055365] rcu: Hierarchical SRCU implementation. [ 0.057477] smp: Bringing up secondary CPUs ... [ 0.059023] x86: Booting SMP configuration: [ 0.060011] .... node #0, CPUs: #1 #2 #3 [ 0.065260] smp: Brought up 1 node, 4 CPUs [ 0.067009] smpboot: Max logical packages: 1 [ 0.068015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131195] node 0 deferred pages initialised in 59ms [ 0.134006] devtmpfs: initialized [ 0.135228] x86/mm: Memory block size: 128MB [ 0.138483] gcov: version magic: 0x41383552 [ 0.140214] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142253] pinctrl core: initialized pinctrl subsystem [ 0.143831] [ 0.144000] ************************************************************* [ 0.144019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145019] ** ** [ 0.146018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147020] ** ** [ 0.148015] ** This means that this kernel is built to expose internal ** [ 0.149021] ** IOMMU data structures, which may compromise security on ** [ 0.150021] ** your system. ** [ 0.151033] ** ** [ 0.152019] ** If you see this message and you are not debugging the ** [ 0.153013] ** kernel, report this immediately to your vendor! ** [ 0.154023] ** ** [ 0.155747] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156024] ************************************************************* [ 0.157751] NET: Registered protocol family 16 [ 0.159445] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.162085] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165136] cpuidle: using governor menu [ 0.169973] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176577] PCI: Using configuration type 1 for base access [ 0.183149] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.199099] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206047] cryptd: max_cpu_qlen set to 1000 [ 0.211202] ACPI: Added _OSI(Module Device) [ 0.212000] ACPI: Added _OSI(Processor Device) [ 0.212022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222399] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235720] ACPI: Interpreter enabled [ 0.238093] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241016] ACPI: Using IOAPIC for interrupt routing [ 0.244055] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250780] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274782] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.278079] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.299287] acpiphp: Slot [2] registered [ 0.301156] acpiphp: Slot [5] registered [ 0.304144] acpiphp: Slot [6] registered [ 0.306167] acpiphp: Slot [7] registered [ 0.310195] acpiphp: Slot [8] registered [ 0.316000] acpiphp: Slot [9] registered [ 0.319140] acpiphp: Slot [10] registered [ 0.322335] acpiphp: Slot [3] registered [ 0.324172] acpiphp: Slot [4] registered [ 0.329133] acpiphp: Slot [11] registered [ 0.331123] acpiphp: Slot [12] registered [ 0.336222] acpiphp: Slot [13] registered [ 0.341325] acpiphp: Slot [14] registered [ 0.344677] acpiphp: Slot [15] registered [ 0.348194] acpiphp: Slot [16] registered [ 0.356190] acpiphp: Slot [17] registered [ 0.361183] acpiphp: Slot [18] registered [ 0.370172] acpiphp: Slot [19] registered [ 0.379159] acpiphp: Slot [20] registered [ 0.386149] acpiphp: Slot [21] registered [ 0.390182] acpiphp: Slot [22] registered [ 0.391144] acpiphp: Slot [23] registered [ 0.393134] acpiphp: Slot [24] registered [ 0.398132] acpiphp: Slot [25] registered [ 0.401119] acpiphp: Slot [26] registered [ 0.406142] acpiphp: Slot [27] registered [ 0.409152] acpiphp: Slot [28] registered [ 0.419169] acpiphp: Slot [29] registered [ 0.422329] acpiphp: Slot [30] registered [ 0.425160] acpiphp: Slot [31] registered [ 0.430144] PCI host bridge to bus 0000:00 [ 0.434030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.439022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.444031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.448026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.455037] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.462036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.467271] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.474674] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.490814] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.502659] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.508055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.512036] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.514041] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.518022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.519000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.522170] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.523041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.526869] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.531800] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.545016] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.549017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.553993] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.564017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.574017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.594032] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.615245] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.646025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.663027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.702026] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.723012] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.731024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.742023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.760000] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.773918] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.778013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.784043] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.799015] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.808392] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.813015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.819014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.831014] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.840877] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.849015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.856014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.873943] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.887464] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.889477] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.892472] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.895702] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.901542] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.912123] iommu: Default domain type: Passthrough [ 0.915321] SCSI subsystem initialized [ 0.917167] ACPI: bus type USB registered [ 0.919082] usbcore: registered new interface driver usbfs [ 0.921103] usbcore: registered new interface driver hub [ 0.924057] usbcore: registered new device driver usb [ 0.926171] pps_core: LinuxPPS API ver. 1 registered [ 0.928012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.932067] PTP clock support registered [ 0.939162] EDAC MC: Ver: 3.0.0 [ 0.942188] PCI: Using ACPI for IRQ routing [ 0.952168] NetLabel: Initializing [ 0.957024] NetLabel: domain hash size = 128 [ 0.959020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.961105] NetLabel: unlabeled traffic allowed by default [ 0.965135] vgaarb: loaded [ 0.972041] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.977025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.993000] clocksource: Switched to clocksource kvm-clock [ 1.118739] VFS: Disk quotas dquot_6.6.0 [ 1.120331] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.123255] *** VALIDATE ramfs *** [ 1.124648] *** VALIDATE hugetlbfs *** [ 1.126228] pnp: PnP ACPI init [ 1.128882] pnp: PnP ACPI: found 6 devices [ 1.144806] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.149342] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.157361] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.164014] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.167277] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.170261] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.173985] NET: Registered protocol family 2 [ 1.182825] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.203188] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.207865] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.244092] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.258210] TCP: Hash tables configured (established 65536 bind 65536) [ 1.265865] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.275494] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.279537] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.287287] NET: Registered protocol family 1 [ 1.290648] RPC: Registered named UNIX socket transport module. [ 1.293194] RPC: Registered udp transport module. [ 1.294887] RPC: Registered tcp transport module. [ 1.298059] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.302091] NET: Registered protocol family 44 [ 1.304066] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.306651] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.308293] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.310221] PCI: CLS 0 bytes, default 64 [ 1.313547] Unpacking initramfs... [ 3.396819] debug: unmapping init [mem 0xffff9882bcc54000-0xffff9882bffbffff] [ 3.409926] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.414630] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.418974] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.583365] Initialise system trusted keyrings [ 4.586021] Key type blacklist registered [ 4.587437] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.596032] zbud: loaded [ 4.599624] *** VALIDATE nfs *** [ 4.600990] *** VALIDATE nfs4 *** [ 4.602930] pstore: using deflate compression [ 4.606696] Platform Keyring initialized [ 4.782358] NET: Registered protocol family 38 [ 4.786823] Key type asymmetric registered [ 4.795542] Asymmetric key parser 'x509' registered [ 4.800351] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.814973] io scheduler mq-deadline registered [ 4.819363] io scheduler kyber registered [ 4.821592] io scheduler bfq registered [ 4.823542] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.845800] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.851368] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.855227] ACPI: Power Button [PWRF] [ 5.035798] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.217361] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.540139] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.871305] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 6.317331] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.360908] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.418879] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.423583] Non-volatile memory driver v1.3 [ 6.426114] Linux agpgart interface v0.103 [ 6.464535] virtio_blk virtio1: [vda] 133056 512-byte logical blocks (68.1 MB/65.0 MiB) [ 6.467717] vda: detected capacity change from 0 to 68124672 [ 6.721573] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.731740] vdb: detected capacity change from 0 to 1073741824 [ 6.765756] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.768745] vdc: detected capacity change from 0 to 2621440000 [ 6.785731] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.788591] vdd: detected capacity change from 0 to 2621440000 [ 6.814267] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.820078] vde: detected capacity change from 0 to 4294967296 [ 6.851565] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.855076] vdf: detected capacity change from 0 to 4294967296 [ 6.869279] libphy: Fixed MDIO Bus: probed [ 6.897828] usbcore: registered new interface driver usbserial_generic [ 6.900387] usbserial: USB Serial support registered for generic [ 6.903363] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.908277] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.912992] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.918666] mousedev: PS/2 mouse device common for all mice [ 6.921171] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.921844] rtc_cmos 00:05: RTC can wake from S4 [ 6.950783] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.951995] rtc_cmos 00:05: registered as rtc0 [ 7.161212] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.198602] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.213027] intel_pstate: CPU model not supported [ 7.227128] hid: raw HID events driver (C) Jiri Kosina [ 7.244130] usbcore: registered new interface driver usbhid [ 7.246067] usbhid: USB HID core driver [ 7.247650] drop_monitor: Initializing network drop monitor service [ 7.249961] Initializing XFRM netlink socket [ 7.251709] NET: Registered protocol family 10 [ 7.264753] Segment Routing with IPv6 [ 7.266302] NET: Registered protocol family 17 [ 7.275425] mpls_gso: MPLS GSO support [ 7.911532] RAS: Correctable Errors collector initialized. [ 7.915345] AVX version of gcm_enc/dec engaged. [ 7.916916] AES CTR mode by8 optimization enabled [ 8.149929] sched_clock: Marking stable (8149909016, 0)->(9607391197, -1457482181) [ 8.160376] registered taskstats version 1 [ 8.170696] Loading compiled-in X.509 certificates [ 8.175921] zswap: loaded using pool lzo/zbud [ 8.229255] Key type big_key registered [ 8.246565] Key type encrypted registered [ 8.248270] ima: No TPM chip found, activating TPM-bypass! [ 8.250373] ima: Allocated hash algorithm: sha1 [ 8.252154] ima: No architecture policies found [ 8.253910] evm: Initialising EVM extended attributes: [ 8.255863] evm: security.selinux [ 8.257268] evm: security.ima [ 8.258334] evm: security.capability [ 8.259605] evm: HMAC attrs: 0x1 [ 8.262232] rtc_cmos 00:05: setting system clock to 2025-09-02 10:33:54 UTC (1756809234) [ 8.269663] debug: unmapping init [mem 0xffffffff94603000-0xffffffff947fffff] [ 8.278438] debug: unmapping init [mem 0xffffffff93382000-0xffffffff93658fff] [ 8.289206] Write protecting the kernel read-only data: 28672k [ 8.298278] debug: unmapping init [mem 0xffffffff91a03000-0xffffffff91bfffff] [ 8.304073] debug: unmapping init [mem 0xffffffff92314000-0xffffffff923fffff] [ 8.391511] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.415253] systemd[1]: Detected virtualization kvm. [ 8.417039] systemd[1]: Detected architecture x86-64. [ 8.419955] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.463760] systemd[1]: No hostname configured. [ 8.466334] systemd[1]: Set hostname to . [ 8.474500] random: systemd: uninitialized urandom read (16 bytes read) [ 8.478996] systemd[1]: Initializing machine ID from random generator. [ 9.012189] random: systemd: uninitialized urandom read (16 bytes read) [ 9.014542] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 9.021636] random: systemd: uninitialized urandom read (16 bytes read) [ 9.025587] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 9.033607] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.039551] device-mapper: uevent: version 1.0.3 [ 11.042391] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 12.621434] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.806881] random: fast init done [ 13.455808] scsi host0: ata_piix [ 13.478896] scsi host1: ata_piix [ 13.482457] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 13.494796] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.289831] hrtimer: interrupt took 5314609 ns [ 18.769489] random: crng init done [ 18.771644] random: 7 urandom warning(s) missed due to ratelimiting [ 21.101481] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.507812] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.612851] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.313469] SELinux: Disabled at runtime. [ 26.463746] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.481933] systemd[1]: Detected virtualization kvm. [ 26.486582] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.942569] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.958384] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.978847] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.009039] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.018088] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.044809] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.068392] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 28.722896] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 30.645117] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.373870] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.745469] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.568863] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.602820] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 36.778039] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (9s / no limit)[ 37.286206] NFS: Registering the id_resolver key type [ 37.288028] Key type id_resolver registered [ 37.289596] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg106-server login: [ 73.749147] libcfs: loading out-of-tree module taints kernel. [ 73.761300] Key type ._llcrypt registered [ 73.763023] Key type .llcrypt registered [ 73.820676] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 82.669422] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 83.388105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 83.393824] alg: No test for adler32 (adler32-zlib) [ 84.413422] Lustre: Lustre: Build Version: 2.16.58_1_g7af8a00 [ 84.715250] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 86.320339] Key type lgssc registered [ 87.059535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.383640] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 97.063027] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 101.990703] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.343810] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 108.759435] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.108660] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 120.647390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 120.669220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 120.679892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 121.776742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 121.791337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 121.832105] Lustre: lustre-MDT0000: new disk, initializing [ 121.863700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 121.873425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 123.326796] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.986030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 129.018684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 129.049863] Lustre: 6492:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 129.062705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 129.064225] Lustre: Skipped 1 previous similar message [ 129.099996] Lustre: lustre-MDT0001: new disk, initializing [ 129.121837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 129.131910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 129.136241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 130.472493] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 132.785575] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 136.369158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.396942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 136.505564] Lustre: lustre-OST0000: new disk, initializing [ 136.507403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 136.525989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.027599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 138.030943] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 138.053115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 138.594582] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.461581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.484024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 144.524649] Lustre: lustre-OST0001: new disk, initializing [ 144.526262] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 144.543254] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 146.291894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 146.296637] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 146.317202] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 146.723531] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 152.955606] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 159.643346] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 166.903144] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing check_logdir /tmp/testlogs/ [ 168.801713] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing yml_node [ 170.676612] Lustre: DEBUG MARKER: Client: 2.16.58.1 [ 171.701640] Lustre: DEBUG MARKER: MDS: 2.16.58.1 [ 172.751041] Lustre: DEBUG MARKER: OSS: 2.16.58.1 [ 173.359519] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Sep 2 06:36:38 EDT 2025 [ 181.210270] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 182.126912] Lustre: DEBUG MARKER: === replay-single: start setup 06:36:47 (1756809407) === [ 183.820154] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing check_config_client /mnt/lustre [ 192.689968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 194.252777] Lustre: 13169:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 195.634813] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.753768] Lustre: DEBUG MARKER: === replay-single: finish setup 06:37:03 (1756809423) === [ 198.476421] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 06:37:03 (1756809423) [ 201.702452] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 202.547435] Lustre: Failing over lustre-MDT0000 [ 202.617813] LustreError: 14038:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 202.649419] Lustre: server umount lustre-MDT0000 complete [ 202.720865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 202.725053] Lustre: Skipped 1 previous similar message [ 207.842141] LustreError: 6502:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 207.854379] LustreError: 6502:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 9 previous similar messages [ 211.974108] LustreError: 9281:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 217.091962] LustreError: 9281:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 217.101956] LustreError: 9281:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 219.065977] LDISKFS-fs (dm-0): recovery complete [ 219.069556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 219.094757] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 219.208570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.601640] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.213229] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 224.229484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 224.255753] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 226.161615] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 227.146712] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 231.985930] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 06:37:37 (1756809457) [ 233.062608] Lustre: Failing over lustre-OST0000 [ 233.099123] LustreError: 15471:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 233.104022] LustreError: 15471:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 233.139468] Lustre: server umount lustre-OST0000 complete [ 234.468326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 234.469911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 234.473609] LustreError: 8382:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 234.478126] Lustre: Skipped 3 previous similar messages [ 234.489708] LustreError: 8382:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 239.585469] LustreError: 8382:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 239.592135] LustreError: 8382:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 247.045203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 247.138221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 247.811198] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 248.693350] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 248.693786] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 248.707215] Lustre: Skipped 3 previous similar messages [ 249.693306] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.415350] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 254.202760] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 259.010921] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 06:38:04 (1756809484) [ 262.397802] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 263.371686] Lustre: Failing over lustre-MDT0000 [ 263.479924] LustreError: 17175:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 263.482278] LustreError: 17175:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 263.516461] Lustre: server umount lustre-MDT0000 complete [ 265.184934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 265.188800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.194401] LustreError: 6502:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 265.202683] LustreError: 6502:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 267.747110] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 267.753638] Lustre: Skipped 1 previous similar message [ 279.087281] LDISKFS-fs (dm-0): recovery complete [ 279.089923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 279.189269] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 279.312186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 281.425581] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 282.604364] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 282.608408] Lustre: lustre-MDT0000: Denying connection for new client 3e7b59d6-5fa0-476f-bb76-e0763a4cdd32 (at 192.168.201.6@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 284.641801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 284.644058] Lustre: Skipped 1 previous similar message [ 287.748261] Lustre: lustre-MDT0000: Denying connection for new client 3e7b59d6-5fa0-476f-bb76-e0763a4cdd32 (at 192.168.201.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 287.754705] Lustre: Skipped 1 previous similar message [ 292.870196] Lustre: lustre-MDT0000: Denying connection for new client 3e7b59d6-5fa0-476f-bb76-e0763a4cdd32 (at 192.168.201.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 297.988457] Lustre: lustre-MDT0000: Denying connection for new client 3e7b59d6-5fa0-476f-bb76-e0763a4cdd32 (at 192.168.201.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 303.108672] Lustre: lustre-MDT0000: Denying connection for new client 3e7b59d6-5fa0-476f-bb76-e0763a4cdd32 (at 192.168.201.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:48 [ 313.348244] Lustre: lustre-MDT0000: Denying connection for new client 3e7b59d6-5fa0-476f-bb76-e0763a4cdd32 (at 192.168.201.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 313.355550] Lustre: Skipped 1 previous similar message [ 333.837309] Lustre: lustre-MDT0000: Denying connection for new client 3e7b59d6-5fa0-476f-bb76-e0763a4cdd32 (at 192.168.201.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 333.853079] Lustre: Skipped 3 previous similar messages [ 352.000587] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 352.005405] Lustre: 17826:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client f3e6091d-94d6-45ac-beda-d6e49db3da41@ [ 352.010790] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 352.020710] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 352.021310] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 352.029118] Lustre: Skipped 2 previous similar messages [ 352.060610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 352.062519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 360.795944] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 06:39:45 (1756809585) [ 365.675456] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 366.981439] Lustre: Failing over lustre-MDT0000 [ 367.079111] LustreError: 18887:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 367.082308] LustreError: 18887:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 367.131610] Lustre: server umount lustre-MDT0000 complete [ 367.584881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 367.588333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 367.592753] Lustre: Skipped 1 previous similar message [ 367.597928] LustreError: 6503:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.609852] LustreError: 6503:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 12 previous similar messages [ 385.371157] LDISKFS-fs (dm-0): recovery complete [ 385.374235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.448632] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 385.591736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.318802] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 389.808795] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 389.814347] Lustre: lustre-MDT0000: Denying connection for new client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 389.822395] Lustre: Skipped 3 previous similar messages [ 390.634116] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 456.707069] Lustre: lustre-MDT0000: Denying connection for new client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:02 [ 456.713892] Lustre: Skipped 12 previous similar messages [ 459.000153] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 459.002497] Lustre: 19538:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 3e7b59d6-5fa0-476f-bb76-e0763a4cdd32@ [ 459.009167] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 459.017265] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 459.017350] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 459.025454] Lustre: Skipped 2 previous similar messages [ 459.039083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 459.039083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 466.718861] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 06:41:32 (1756809692) [ 470.625143] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 471.649874] Lustre: Failing over lustre-MDT0000 [ 471.732331] LustreError: 20595:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 471.736538] LustreError: 20595:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 471.785757] Lustre: server umount lustre-MDT0000 complete [ 472.081760] LustreError: 18188:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.088418] LustreError: 18188:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 12 previous similar messages [ 472.546415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.556623] Lustre: Skipped 6 previous similar messages [ 487.217685] LDISKFS-fs (dm-0): recovery complete [ 487.220043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.266608] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 487.350363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.426924] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 489.099145] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 492.517000] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 492.541546] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 492.564852] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 492.564869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 494.316980] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 495.132369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 500.010468] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 06:42:05 (1756809725) [ 504.044148] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 505.136536] Lustre: Failing over lustre-MDT0000 [ 505.367273] LustreError: 22416:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 505.372107] LustreError: 22416:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 505.423984] Lustre: server umount lustre-MDT0000 complete [ 507.873822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.886065] Lustre: Skipped 3 previous similar messages [ 507.890397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 521.643941] LDISKFS-fs (dm-0): recovery complete [ 521.646588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.728297] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.268038] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 524.105854] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 527.335253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 527.337610] Lustre: Skipped 3 previous similar messages [ 527.382465] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 527.411370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 527.411370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 529.431618] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 530.356081] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 535.384288] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 06:42:40 (1756809760) [ 539.080949] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 539.976265] Lustre: Failing over lustre-MDT0000 [ 540.226211] LustreError: 24238:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 540.228828] LustreError: 24238:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 540.254710] Lustre: server umount lustre-MDT0000 complete [ 542.689239] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.692613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 542.693138] LustreError: 6503:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 542.693147] LustreError: 6503:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 29 previous similar messages [ 542.696939] Lustre: Skipped 2 previous similar messages [ 556.175328] LDISKFS-fs (dm-0): recovery complete [ 556.178568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.238402] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.094023] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 559.000863] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 561.661480] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 561.680538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 561.680570] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 563.185871] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 564.697501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 568.909450] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 06:43:14 (1756809794) [ 572.355133] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 573.280905] Lustre: Failing over lustre-MDT0000 [ 573.416121] LustreError: 26059:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 573.418489] LustreError: 26059:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 573.454672] Lustre: server umount lustre-MDT0000 complete [ 576.993669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 589.488420] LDISKFS-fs (dm-0): recovery complete [ 589.490748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 589.565880] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.677143] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 594.916389] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 594.920439] Lustre: Skipped 7 previous similar messages [ 594.959779] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 594.980916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 594.981034] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 596.566167] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 597.252617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 601.923032] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 06:43:47 (1756809827) [ 605.908517] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 607.082743] Lustre: Failing over lustre-MDT0000 [ 607.289076] Lustre: server umount lustre-MDT0000 complete [ 610.273169] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.280204] Lustre: Skipped 8 previous similar messages [ 610.282421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.607294] LDISKFS-fs (dm-0): recovery complete [ 623.609866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 623.670933] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.769488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 623.772583] Lustre: Skipped 3 previous similar messages [ 623.798107] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 625.590503] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 625.669708] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 625.678983] Lustre: Skipped 1 previous similar message [ 629.272436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 629.272598] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 631.060395] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 631.939640] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 636.549959] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 06:44:21 (1756809861) [ 637.038207] Lustre: *** cfs_fail_loc=13b, val=315*** [ 637.040567] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 637.043027] LustreError: 6501:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff988332c99180 x1842148069729536/t38654705666(0) o35->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:374/0 lens 392/456 e 0 to 0 dl 1756809874 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 641.680612] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 642.541614] Lustre: Failing over lustre-MDT0000 [ 642.694597] LustreError: 29754:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 642.698021] LustreError: 29754:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 642.754553] Lustre: server umount lustre-MDT0000 complete [ 644.579413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 658.806541] LDISKFS-fs (dm-0): recovery complete [ 658.809397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.972124] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 660.854332] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 664.060897] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 664.065181] Lustre: Skipped 1 previous similar message [ 664.066087] Lustre: 6501:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff988332e47480 x1842148069729536/t38654705666(0) o35->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:401/0 lens 392/456 e 0 to 0 dl 1756809901 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 664.088236] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 664.088421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 665.898128] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 666.834209] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 671.982944] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 06:44:57 (1756809897) [ 676.082503] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 677.133203] Lustre: Failing over lustre-MDT0000 [ 677.314849] Lustre: server umount lustre-MDT0000 complete [ 679.393978] LustreError: 6503:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 679.401351] LustreError: 6503:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 62 previous similar messages [ 693.498399] LDISKFS-fs (dm-0): recovery complete [ 693.500242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.568035] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 693.572295] LustreError: Skipped 1 previous similar message [ 693.681777] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 695.715384] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 697.347109] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 697.352200] Lustre: Skipped 1 previous similar message [ 698.893408] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 698.893516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 700.650870] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 701.482434] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 706.049314] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 06:45:31 (1756809931) [ 709.876448] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 710.356777] Lustre: *** cfs_fail_loc=114, val=0*** [ 711.732095] Lustre: Failing over lustre-MDT0000 [ 711.967631] Lustre: server umount lustre-MDT0000 complete [ 714.209574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 714.214644] LustreError: Skipped 1 previous similar message [ 727.896407] LDISKFS-fs (dm-0): recovery complete [ 727.898430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.083060] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 729.857843] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 733.155535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 733.159638] Lustre: Skipped 15 previous similar messages [ 733.192743] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 733.192816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 734.729587] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 735.495696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 740.803419] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 06:46:06 (1756809966) [ 745.041441] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 745.518142] Lustre: *** cfs_fail_loc=128, val=0*** [ 747.038284] Lustre: Failing over lustre-MDT0000 [ 747.223530] Lustre: server umount lustre-MDT0000 complete [ 748.513537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 748.522544] Lustre: Skipped 15 previous similar messages [ 763.065522] LDISKFS-fs (dm-0): recovery complete [ 763.067991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.254207] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 765.173626] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 768.544764] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 768.544795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 770.228565] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 771.087311] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 776.164973] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 06:46:41 (1756810001) [ 780.292311] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 781.345049] Lustre: Failing over lustre-MDT0000 [ 781.503747] LustreError: 37234:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 781.505831] LustreError: 37234:0:(obd_class.h:479:obd_check_dev()) Skipped 31 previous similar messages [ 781.540232] Lustre: server umount lustre-MDT0000 complete [ 797.169117] LDISKFS-fs (dm-0): recovery complete [ 797.171219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.321152] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 799.090981] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 802.890708] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 802.896195] Lustre: Skipped 3 previous similar messages [ 802.919252] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 802.919296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 804.779085] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 805.653168] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 811.965938] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 06:47:16 (1756810036) [ 821.886081] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 825.366617] Lustre: Failing over lustre-MDT0000 [ 826.113493] Lustre: server umount lustre-MDT0000 complete [ 844.708397] Lustre: 3634:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810054/real 1756810054] req@ffff98833d80dc00 x1842148079516032/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1756810070 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 844.758321] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 844.778889] LustreError: Skipped 3 previous similar messages [ 853.527498] LDISKFS-fs (dm-0): recovery complete [ 853.538539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.629432] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 856.072577] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 856.098388] Lustre: Skipped 3 previous similar messages [ 860.724608] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 860.914986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 860.915541] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 869.966643] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 872.083129] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 881.611755] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 06:48:26 (1756810106) [ 888.582857] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 896.893173] Lustre: Failing over lustre-MDT0000 [ 896.998383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 897.006803] LustreError: Skipped 2 previous similar messages [ 897.020902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.114576] Lustre: server umount lustre-MDT0000 complete [ 917.490394] LDISKFS-fs (dm-0): recovery complete [ 917.494212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.759515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.763787] Lustre: Skipped 6 previous similar messages [ 917.795907] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 920.788205] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 926.316359] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 926.316435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 929.090739] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 930.496613] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 951.345993] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 06:49:36 (1756810176) [ 956.977154] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 958.698395] Lustre: Failing over lustre-MDT0000 [ 958.907794] Lustre: server umount lustre-MDT0000 complete [ 958.947554] LustreError: 8408:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 958.956619] LustreError: 8408:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 114 previous similar messages [ 979.549921] LDISKFS-fs (dm-0): recovery complete [ 979.553538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 983.490173] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 985.193562] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 985.195157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 989.870986] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 991.071442] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 999.999955] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 06:50:24 (1756810224) [ 1006.119923] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1007.869241] Lustre: Failing over lustre-MDT0000 [ 1008.092988] Lustre: server umount lustre-MDT0000 complete [ 1010.657933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1010.688639] Lustre: Skipped 16 previous similar messages [ 1026.018345] Lustre: 3636:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810236/real 1756810236] req@ffff98832f8fd180 x1842148079692160/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1756810252 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1028.559749] LDISKFS-fs (dm-0): recovery complete [ 1028.563179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.256472] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff988332fda300 x1842148079699584/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1036.459269] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1036.465321] Lustre: Skipped 1 previous similar message [ 1039.525787] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1041.899604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1041.911619] Lustre: Skipped 23 previous similar messages [ 1042.000482] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 1042.004093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 1045.668230] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1046.860612] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1053.440711] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 06:51:18 (1756810278) [ 1058.914336] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1060.241617] Lustre: Failing over lustre-MDT0000 [ 1060.383155] LustreError: 46401:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1060.386142] LustreError: 46401:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 1060.454740] Lustre: server umount lustre-MDT0000 complete [ 1078.745372] Lustre: 3633:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810288/real 1756810288] req@ffff9882072ba300 x1842148079722112/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1756810304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1081.022451] LDISKFS-fs (dm-0): recovery complete [ 1081.028259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.992832] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff988332d18a80 x1842148079730560/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1092.727651] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1094.720798] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1094.729832] Lustre: Skipped 4 previous similar messages [ 1094.783913] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1094.786403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1099.670782] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1101.277519] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1109.102578] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 06:52:13 (1756810333) [ 1115.967734] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1117.951694] Lustre: Failing over lustre-MDT0000 [ 1118.196113] Lustre: server umount lustre-MDT0000 complete [ 1136.610038] Lustre: 3635:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810346/real 1756810346] req@ffff9882072baa00 x1842148079756288/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1756810362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1136.628586] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1136.640384] LustreError: Skipped 4 previous similar messages [ 1138.346651] LDISKFS-fs (dm-0): recovery complete [ 1138.350564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.824612] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff988332cce300 x1842148079764992/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1146.884433] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1146.903360] Lustre: Skipped 4 previous similar messages [ 1148.999870] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1151.592189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1151.592394] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1154.803446] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1156.121702] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1162.783205] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 06:53:07 (1756810387) [ 1169.045245] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1170.916699] Lustre: Failing over lustre-MDT0000 [ 1171.123959] Lustre: server umount lustre-MDT0000 complete [ 1171.939284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1171.944531] LustreError: Skipped 3 previous similar messages [ 1188.320190] Lustre: 3635:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810398/real 1756810398] req@ffff98833338a300 x1842148079789184/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1756810414 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1191.033688] LDISKFS-fs (dm-0): recovery complete [ 1191.037269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.784179] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1198.790837] Lustre: Skipped 2 previous similar messages [ 1202.141870] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1204.288390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1204.292748] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1208.767218] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1210.227372] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1217.315203] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 06:54:02 (1756810442) [ 1224.188706] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1226.138591] Lustre: Failing over lustre-MDT0000 [ 1226.425585] Lustre: server umount lustre-MDT0000 complete [ 1246.177436] Lustre: 3634:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810456/real 1756810456] req@ffff988206e51880 x1842148079822592/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1756810472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1246.292590] LDISKFS-fs (dm-0): recovery complete [ 1246.304218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.351602] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1262.167050] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1262.168457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1267.369972] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1269.038878] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1276.657798] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 06:55:01 (1756810501) [ 1283.797426] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1285.750734] Lustre: Failing over lustre-MDT0000 [ 1286.037025] Lustre: server umount lustre-MDT0000 complete [ 1303.507422] Lustre: 3635:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810513/real 1756810513] req@ffff98833af8e300 x1842148079855744/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1756810529 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1306.810965] LDISKFS-fs (dm-0): recovery complete [ 1306.814655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1313.760965] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98834184c700 x1842148079863296/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1317.757215] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1319.561731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1319.566568] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1323.888828] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1325.334228] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1332.593460] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 06:55:57 (1756810557) [ 1338.892596] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1340.532021] Lustre: Failing over lustre-MDT0000 [ 1340.697255] Lustre: server umount lustre-MDT0000 complete [ 1360.681779] LDISKFS-fs (dm-0): recovery complete [ 1360.687976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.125676] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1366.094855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1366.096291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1370.508115] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1371.900767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1378.400445] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 06:56:43 (1756810603) [ 1384.497801] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1386.100625] Lustre: Failing over lustre-MDT0000 [ 1386.370708] Lustre: server umount lustre-MDT0000 complete [ 1406.708503] LDISKFS-fs (dm-0): recovery complete [ 1406.712893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.948907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 1406.963374] Lustre: Skipped 1 previous similar message [ 1410.411910] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1412.122921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:865) [ 1412.131113] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:865) [ 1416.107762] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1417.429868] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1424.246091] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 06:57:29 (1756810649) [ 1430.155560] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1431.495113] Lustre: Failing over lustre-MDT0000 [ 1431.701866] Lustre: server umount lustre-MDT0000 complete [ 1447.907920] Lustre: 3634:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810658/real 1756810658] req@ffff988333388380 x1842148079945344/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1756810674 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1452.111772] LDISKFS-fs (dm-0): recovery complete [ 1452.115361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.436174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.440351] Lustre: Skipped 9 previous similar messages [ 1458.482807] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1458.486471] Lustre: Skipped 4 previous similar messages [ 1462.044150] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1463.899444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:835 to 0x280000401:897) [ 1463.905603] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 1468.188765] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1469.552217] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1475.989069] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 06:58:21 (1756810701) [ 1481.774085] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1483.469240] Lustre: Failing over lustre-MDT0000 [ 1483.691428] Lustre: server umount lustre-MDT0000 complete [ 1484.259854] LustreError: 6498:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.268275] LustreError: 6498:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 289 previous similar messages [ 1502.002432] LDISKFS-fs (dm-0): recovery complete [ 1502.007712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.858449] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xed33cf38b8d4bac [ 1512.638745] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1515.591542] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1515.592497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1518.039925] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1519.239344] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1526.100921] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 06:59:11 (1756810751) [ 1531.660391] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1533.243416] Lustre: Failing over lustre-MDT0000 [ 1533.436950] Lustre: server umount lustre-MDT0000 complete [ 1535.970469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.977062] Lustre: Skipped 41 previous similar messages [ 1552.079540] LDISKFS-fs (dm-0): recovery complete [ 1552.082515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1555.009541] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1557.479624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1557.487232] Lustre: Skipped 40 previous similar messages [ 1557.560819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 1557.561618] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 1560.636256] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1561.737716] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1567.649597] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 06:59:52 (1756810792) [ 1572.754431] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1574.172274] Lustre: Failing over lustre-MDT0000 [ 1574.354718] LustreError: 64648:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1574.358780] LustreError: 64648:0:(obd_class.h:479:obd_check_dev()) Skipped 79 previous similar messages [ 1574.403394] Lustre: server umount lustre-MDT0000 complete [ 1593.001349] LDISKFS-fs (dm-0): recovery complete [ 1593.008562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.889944] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1598.518039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 1598.518274] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:993) [ 1601.553761] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1602.818825] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1609.510530] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 07:00:34 (1756810834) [ 1614.815092] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1616.334596] Lustre: Failing over lustre-MDT0000 [ 1616.516452] Lustre: server umount lustre-MDT0000 complete [ 1634.007648] LDISKFS-fs (dm-0): recovery complete [ 1634.013300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.173955] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1639.482933] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1639.490286] Lustre: Skipped 10 previous similar messages [ 1639.527090] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1639.527384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1642.866814] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1644.017685] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1650.313848] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 07:01:15 (1756810875) [ 1656.576856] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1658.114901] Lustre: Failing over lustre-MDT0000 [ 1658.287416] Lustre: server umount lustre-MDT0000 complete [ 1676.258292] Lustre: 3633:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756810886/real 1756810886] req@ffff98833ab1ea00 x1842148080091264/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1756810902 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1676.271608] Lustre: 3633:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1676.276470] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.289493] LustreError: Skipped 10 previous similar messages [ 1677.642547] LDISKFS-fs (dm-0): recovery complete [ 1677.645733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.498370] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98833aa04e00 x1842148080099840/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1687.528497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1687.534909] Lustre: Skipped 10 previous similar messages [ 1689.702296] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1692.243744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1692.244110] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1696.038489] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1697.492469] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1703.911254] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 07:02:08 (1756810928) [ 1709.557475] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1711.010736] Lustre: Failing over lustre-MDT0000 [ 1711.206956] Lustre: server umount lustre-MDT0000 complete [ 1712.616175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1712.622505] LustreError: Skipped 9 previous similar messages [ 1729.980578] LDISKFS-fs (dm-0): recovery complete [ 1729.984079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.159373] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1743.945774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1089) [ 1743.951659] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1089) [ 1746.830700] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1747.903513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1754.005193] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 07:02:59 (1756810979) [ 1756.659729] Lustre: 71608:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 4f47911a-29dc-49cd-abfa-4405de2af7b6 at adminstrative request [ 1761.463526] Lustre: Failing over lustre-MDT0000 [ 1761.666820] Lustre: server umount lustre-MDT0000 complete [ 1777.773855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.632701] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1783.307674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1783.307684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1059 to 0x2c0000401:1121) [ 1785.785195] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1787.092953] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1790.547434] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1795.195584] Lustre: DEBUG MARKER: before 3128, after 3128 [ 1799.272804] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 07:03:44 (1756811024) [ 1799.999477] Lustre: 73453:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 4f47911a-29dc-49cd-abfa-4405de2af7b6 at adminstrative request [ 1806.097978] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 07:03:51 (1756811031) [ 1811.623176] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1813.053510] Lustre: Failing over lustre-MDT0000 [ 1813.267580] Lustre: server umount lustre-MDT0000 complete [ 1831.416594] LDISKFS-fs (dm-0): recovery complete [ 1831.422202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.349721] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1837.143247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 1837.149482] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 1839.602274] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1840.766774] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1847.191535] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 07:04:32 (1756811072) [ 1852.363432] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1853.844246] Lustre: Failing over lustre-MDT0000 [ 1854.059946] Lustre: server umount lustre-MDT0000 complete [ 1871.187769] LDISKFS-fs (dm-0): recovery complete [ 1871.190752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1874.188582] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1876.519292] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1876.519903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1879.525668] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1880.528190] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1886.385918] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 07:05:11 (1756811111) [ 1891.349274] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1892.588880] Lustre: Failing over lustre-MDT0000 [ 1892.787868] Lustre: server umount lustre-MDT0000 complete [ 1910.325613] LDISKFS-fs (dm-0): recovery complete [ 1910.330037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1912.954867] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1915.933957] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1915.934947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1918.388325] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1919.630486] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1925.552489] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 07:05:50 (1756811150) [ 1930.541454] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1931.837145] Lustre: Failing over lustre-MDT0000 [ 1932.057404] Lustre: server umount lustre-MDT0000 complete [ 1950.639698] LDISKFS-fs (dm-0): recovery complete [ 1950.643604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.035270] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1956.388108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1956.389142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1959.800461] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1961.021946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1966.970147] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 07:06:32 (1756811192) [ 1972.218769] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1973.297217] Lustre: Failing over lustre-MDT0000 [ 1973.495281] Lustre: server umount lustre-MDT0000 complete [ 1992.153977] LDISKFS-fs (dm-0): recovery complete [ 1992.157484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.482533] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1992.493301] Lustre: Skipped 11 previous similar messages [ 1995.237551] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1997.840365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 1997.842376] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 2000.720608] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2001.796853] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2007.653620] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 07:07:12 (1756811232) [ 2012.207335] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2013.505688] Lustre: Failing over lustre-MDT0000 [ 2013.780696] Lustre: server umount lustre-MDT0000 complete [ 2031.716306] LDISKFS-fs (dm-0): recovery complete [ 2031.720137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.519207] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2037.323682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 2037.324696] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 2039.621577] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2040.628276] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2045.916201] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 07:07:51 (1756811271) [ 2050.538316] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2051.834542] Lustre: Failing over lustre-MDT0000 [ 2051.967883] Lustre: server umount lustre-MDT0000 complete [ 2067.936409] Lustre: 3636:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756811278/real 1756811278] req@ffff98833a9efb80 x1842148080346368/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1756811294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2067.954657] Lustre: 3636:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2069.215473] LDISKFS-fs (dm-0): recovery complete [ 2069.217759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2078.183157] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xed33cf38b8d93dc [ 2078.308179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.312748] Lustre: Skipped 13 previous similar messages [ 2080.721762] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2083.880772] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 2083.883098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 2086.304645] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2087.397342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2093.278285] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 07:08:38 (1756811318) [ 2098.175864] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2099.447601] Lustre: Failing over lustre-MDT0000 [ 2099.671875] Lustre: server umount lustre-MDT0000 complete [ 2100.751825] LustreError: 6499:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.766258] LustreError: 6499:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 291 previous similar messages [ 2117.672248] LDISKFS-fs (dm-0): recovery complete [ 2117.675152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.822193] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2123.305414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 2123.305414] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 2125.894620] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2127.059034] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2132.777956] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 07:09:17 (1756811357) [ 2137.195712] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2138.406558] Lustre: Failing over lustre-MDT0000 [ 2138.566282] Lustre: server umount lustre-MDT0000 complete [ 2138.593761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.600300] Lustre: Skipped 55 previous similar messages [ 2156.978224] LDISKFS-fs (dm-0): recovery complete [ 2156.981700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.781105] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2162.663637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2162.668215] Lustre: Skipped 56 previous similar messages [ 2162.719059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 2162.719659] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 2164.905556] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2165.876435] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2171.210961] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 07:09:56 (1756811396) [ 2175.672556] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2176.767763] Lustre: Failing over lustre-MDT0000 [ 2176.907218] LustreError: 90544:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2176.911671] LustreError: 90544:0:(obd_class.h:479:obd_check_dev()) Skipped 111 previous similar messages [ 2176.952252] Lustre: server umount lustre-MDT0000 complete [ 2194.512603] LDISKFS-fs (dm-0): recovery complete [ 2194.516743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2204.640784] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98833a94f480 x1842148080435968/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2207.214293] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2210.315244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 2210.315656] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 2212.024132] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2212.894201] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2218.085641] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 07:10:43 (1756811443) [ 2223.204863] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2224.619533] Lustre: Failing over lustre-MDT0000 [ 2224.779948] Lustre: server umount lustre-MDT0000 complete [ 2242.043802] LDISKFS-fs (dm-0): recovery complete [ 2242.046884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2251.235924] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98833d47f480 x1842148080461824/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2253.801730] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2256.883689] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2256.887733] Lustre: Skipped 13 previous similar messages [ 2256.909217] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 2256.909234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 2258.955150] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2260.086756] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2265.291839] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 07:11:30 (1756811490) [ 2265.944764] Lustre: 93758:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 4f47911a-29dc-49cd-abfa-4405de2af7b6 at adminstrative request [ 2273.058130] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 07:11:38 (1756811498) [ 2277.195971] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2278.366159] Lustre: Failing over lustre-MDT0000 [ 2278.409488] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 2278.576333] Lustre: server umount lustre-MDT0000 complete [ 2286.944188] LDISKFS-fs (dm-0): recovery complete [ 2286.950051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2287.027728] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2287.034849] LustreError: Skipped 13 previous similar messages [ 2287.213387] Lustre: lustre-MDT0000: Aborting client recovery [ 2287.215583] LustreError: 95039:0:(ldlm_lib.c:2935:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2287.219985] Lustre: 95072:0:(ldlm_lib.c:2338:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2287.225436] Lustre: 95072:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 4f47911a-29dc-49cd-abfa-4405de2af7b6@ [ 2287.230832] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2287.236639] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2287.246667] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2287.284868] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 2287.297244] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 2289.977291] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2292.198349] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2305.811562] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 07:12:11 (1756811531) [ 2310.000502] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2311.017161] Lustre: Failing over lustre-MDT0000 [ 2311.187223] Lustre: server umount lustre-MDT0000 complete [ 2312.674937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2312.691856] LustreError: Skipped 11 previous similar messages [ 2319.137369] LDISKFS-fs (dm-0): recovery complete [ 2319.141225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.321680] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2319.358926] Lustre: lustre-MDT0000: Aborting client recovery [ 2319.361219] LustreError: 96885:0:(ldlm_lib.c:2935:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2319.365971] Lustre: 96918:0:(ldlm_lib.c:2338:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2319.371995] Lustre: 96918:0:(ldlm_lib.c:2338:target_recovery_overseer()) Skipped 2 previous similar messages [ 2319.374702] Lustre: 96918:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2319.382877] Lustre: 96918:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2319.388154] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2319.391382] Lustre: 96918:0:(ldlm_lib.c:1850:abort_req_replay_queue()) @@@ aborted: req@ffff98833a951500 x1842148070730240/t0(193273528357) o101->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:546/0 lens 592/0 e 0 to 0 dl 1756811556 ref 1 fl Complete:/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 2319.410930] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 2319.416825] Lustre: lustre-MDT0000: Denying connection for new client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 2 evicted) already passed deadline 38:39 [ 2319.428543] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 2319.473104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1513 to 0x280000401:1537) [ 2319.475977] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1512 to 0x2c0000401:1537) [ 2321.824699] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2324.456696] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2333.244331] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2333.246659] Lustre: Skipped 1 previous similar message [ 2337.264659] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 07:12:42 (1756811562) [ 2341.519611] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2342.529594] Lustre: Failing over lustre-MDT0000 [ 2342.637477] Lustre: server umount lustre-MDT0000 complete [ 2349.932796] LDISKFS-fs (dm-0): recovery complete [ 2349.935744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.048532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2350.053156] Lustre: Skipped 1 previous similar message [ 2350.174635] Lustre: lustre-MDT0000: Aborting client recovery [ 2350.177473] LustreError: 98733:0:(ldlm_lib.c:2935:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2350.184346] Lustre: 98766:0:(ldlm_lib.c:2338:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2350.190498] Lustre: 98766:0:(ldlm_lib.c:2338:target_recovery_overseer()) Skipped 2 previous similar messages [ 2350.195902] Lustre: 98766:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 4f47911a-29dc-49cd-abfa-4405de2af7b6@ [ 2350.204308] Lustre: 98766:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2350.210249] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2350.221219] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 2350.237183] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 2350.281909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1544 to 0x280000401:1569) [ 2350.282079] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1569) [ 2352.628984] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2355.169864] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2368.566916] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 07:13:13 (1756811593) [ 2369.178265] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2369.181977] LustreError: 18188:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff9883012a8380 x1842148070769408/t201863462916(0) o36->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:596/0 lens 512/456 e 0 to 0 dl 1756811606 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 2372.319386] Lustre: Failing over lustre-MDT0000 [ 2372.465536] Lustre: server umount lustre-MDT0000 complete [ 2378.023980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.265285] Lustre: lustre-MDT0000: Aborting client recovery [ 2378.267628] LustreError: 100040:0:(ldlm_lib.c:2935:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2378.271981] Lustre: 100073:0:(ldlm_lib.c:2338:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2378.276437] Lustre: 100073:0:(ldlm_lib.c:2338:target_recovery_overseer()) Skipped 2 previous similar messages [ 2378.282115] Lustre: 100073:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2378.288951] Lustre: 100073:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2378.294251] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2378.299952] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 2378.308394] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 2378.349911] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1601) [ 2378.350592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 2380.767894] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2383.337198] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2397.273938] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 2398.448675] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 07:13:43 (1756811623) [ 2403.641358] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2405.256802] Lustre: Failing over lustre-MDT0000 [ 2405.483729] Lustre: server umount lustre-MDT0000 complete [ 2412.675126] LDISKFS-fs (dm-0): recovery complete [ 2412.680248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2412.963590] Lustre: lustre-MDT0000: Aborting client recovery [ 2412.965765] LustreError: 101986:0:(ldlm_lib.c:2935:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2412.971560] Lustre: 102019:0:(ldlm_lib.c:2338:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2412.975550] Lustre: 102019:0:(ldlm_lib.c:2338:target_recovery_overseer()) Skipped 2 previous similar messages [ 2412.982038] Lustre: 102019:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2412.986300] Lustre: 102019:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2412.989444] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2412.994365] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 2413.006135] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 2413.037394] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1543 to 0x2c0000401:1633) [ 2413.045358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 2415.162288] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2418.145378] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2430.103407] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 07:14:15 (1756811655) [ 2449.352987] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2450.246794] Lustre: Failing over lustre-MDT0000 [ 2450.475883] Lustre: server umount lustre-MDT0000 complete [ 2467.131469] LDISKFS-fs (dm-0): recovery complete [ 2467.135398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2467.844550] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2467.852663] Lustre: Skipped 13 previous similar messages [ 2469.628537] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2472.502260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 2472.502280] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 2474.463676] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2475.358905] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2487.333983] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 07:15:12 (1756811712) [ 2503.663819] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2509.643527] Lustre: Failing over lustre-MDT0000 [ 2509.664793] LustreError: 3633:0:(client.c:1371:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff988204b77b80 x1842148080820352/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 2509.868924] Lustre: server umount lustre-MDT0000 complete [ 2527.400165] LDISKFS-fs (dm-0): recovery complete [ 2527.401890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.028384] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2535.005882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 2535.007555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 2537.355837] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2538.292695] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2551.794831] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 07:16:17 (1756811777) [ 2552.830699] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2553.309141] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2553.313762] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2556.433610] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 07:16:21 (1756811781) [ 2570.939546] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2577.443420] Lustre: Failing over lustre-OST0000 [ 2577.505851] Lustre: server umount lustre-OST0000 complete [ 2594.494137] LDISKFS-fs (dm-2): recovery complete [ 2594.498680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2594.602378] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2594.606800] Lustre: Skipped 23 previous similar messages [ 2597.502211] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2646.625997] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 07:17:52 (1756811872) [ 2650.386496] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2651.734588] Lustre: Failing over lustre-MDT0000 [ 2652.005544] Lustre: server umount lustre-MDT0000 complete [ 2668.703849] LDISKFS-fs (dm-0): recovery complete [ 2668.706497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.789551] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2674.178625] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2674.178699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 2675.852893] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2676.647057] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2690.528075] Lustre: 111113:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756811900/real 1756811900] req@ffff98833a883b80 x1842148081180672/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1756811916 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 2690.538825] Lustre: 111113:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2690.545344] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2690.546288] LustreError: 111113:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2691.278075] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 07:18:36 (1756811916) [ 2691.616460] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 2693.647565] LustreError: 6498:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_race id 701 sleeping [ 2698.720132] LustreError: 6498:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2698.724395] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnecting [ 2699.507815] LustreError: 6498:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_race id 701 sleeping [ 2704.864076] LustreError: 6498:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2704.867652] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnecting [ 2705.718946] LustreError: 6498:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_race id 701 sleeping [ 2711.008360] LustreError: 6498:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2711.014807] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnecting [ 2711.952950] LustreError: 6498:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_race id 701 sleeping [ 2717.152771] LustreError: 6498:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2718.157358] LustreError: 6499:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_race id 701 sleeping [ 2723.297894] LustreError: 6499:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2723.304190] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnecting [ 2723.309407] Lustre: Skipped 1 previous similar message [ 2730.929001] LustreError: 6497:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_race id 701 sleeping [ 2730.933289] LustreError: 6497:0:(ldlm_lib.c:1118:target_handle_connect()) Skipped 1 previous similar message [ 2736.096138] LustreError: 6497:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2736.102981] LustreError: 6497:0:(ldlm_lib.c:1118:target_handle_connect()) Skipped 1 previous similar message [ 2742.240910] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnecting [ 2742.245479] Lustre: Skipped 2 previous similar messages [ 2749.345807] LustreError: 8408:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_race id 701 sleeping [ 2749.350126] LustreError: 8408:0:(ldlm_lib.c:1118:target_handle_connect()) Skipped 2 previous similar messages [ 2754.528312] LustreError: 8408:0:(ldlm_lib.c:1118:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2754.532464] LustreError: 8408:0:(ldlm_lib.c:1118:target_handle_connect()) Skipped 2 previous similar messages [ 2758.943955] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 07:19:44 (1756811984) [ 2759.941452] LustreError: 6499:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2770.435593] Lustre: lustre-MDT0000: Export ffff988329edc000 already connecting from 192.168.201.6@tcp [ 2775.555918] Lustre: lustre-MDT0000: Export ffff988329edc000 already connecting from 192.168.201.6@tcp [ 2780.675441] Lustre: lustre-MDT0000: Export ffff988329edc000 already connecting from 192.168.201.6@tcp [ 2785.796259] Lustre: lustre-MDT0000: Export ffff988329edc000 already connecting from 192.168.201.6@tcp [ 2785.801425] Lustre: Skipped 1 previous similar message [ 2790.919964] Lustre: lustre-MDT0000: Export ffff988329edc000 already connecting from 192.168.201.6@tcp [ 2800.008337] LustreError: 6499:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2800.012716] Lustre: 6499:0:(service.c:2559:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff98833b1a8700 x1842148073410560/t0(0) o38->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:0/0 lens 520/416 e 0 to 0 dl 1756812006 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2801.158345] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnecting [ 2801.163689] Lustre: Skipped 3 previous similar messages [ 2801.167074] LustreError: 6499:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2826.758264] Lustre: lustre-MDT0000: Export ffff988329edc000 already connecting from 192.168.201.6@tcp [ 2826.762678] Lustre: Skipped 1 previous similar message [ 2841.208118] LustreError: 6499:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2841.214111] Lustre: 6499:0:(service.c:2559:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9883385ac380 x1842148073415040/t0(0) o38->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:0/0 lens 520/416 e 0 to 0 dl 1756812047 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2842.115837] LustreError: 18188:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2867.719216] Lustre: lustre-MDT0000: Export ffff988329edc000 already connecting from 192.168.201.6@tcp [ 2867.723569] Lustre: Skipped 2 previous similar messages [ 2882.168346] LustreError: 18188:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2882.174779] Lustre: 18188:0:(service.c:2559:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff988333282a00 x1842148073418240/t0(0) o38->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:0/0 lens 520/416 e 0 to 0 dl 1756812088 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2883.078190] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnecting [ 2883.083646] Lustre: Skipped 1 previous similar message [ 2883.087155] LustreError: 18188:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2908.675364] Lustre: lustre-MDT0000: Export ffff988329edc000 already connecting from 192.168.201.6@tcp [ 2908.678674] Lustre: Skipped 2 previous similar messages [ 2923.136174] LustreError: 18188:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2923.140401] Lustre: 18188:0:(service.c:2559:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff988204b74a80 x1842148073421312/t0(0) o38->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:0/0 lens 520/416 e 0 to 0 dl 1756812129 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2924.035853] LustreError: 6497:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2964.088081] LustreError: 6497:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2964.092941] Lustre: 6497:0:(service.c:2559:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff98833b1a9880 x1842148073424384/t0(0) o38->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:0/0 lens 520/416 e 0 to 0 dl 1756812170 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2964.995494] LustreError: 6497:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2973.632065] LustreError: 6497:0:(ldlm_lib.c:1371:target_handle_connect()) cfs_fail_timeout interrupted [ 2975.505467] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 07:23:20 (1756812200) [ 2978.854842] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2980.552787] Lustre: Failing over lustre-MDT0000 [ 2980.638822] LustreError: 115199:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2980.642740] LustreError: 115199:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages [ 2981.344599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2981.347895] LustreError: Skipped 13 previous similar messages [ 2981.350520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.356519] Lustre: Skipped 45 previous similar messages [ 2981.358763] LustreError: 38239:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.366029] LustreError: 38239:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 188 previous similar messages [ 2982.712653] Lustre: server umount lustre-MDT0000 complete [ 2988.302579] LDISKFS-fs (dm-0): recovery complete [ 2988.304592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.382890] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2988.393047] LustreError: Skipped 7 previous similar messages [ 2988.473628] Lustre: *** cfs_fail_loc=712, val=0*** [ 2988.476667] LustreError: 42056:0:(service.c:1388:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff988333388000 x1842148081332864/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2988.496210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2988.499960] Lustre: Skipped 13 previous similar messages [ 2988.546767] Lustre: lustre-MDT0000: Aborting client recovery [ 2988.550170] LustreError: 115843:0:(ldlm_lib.c:2935:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2988.555132] Lustre: 115876:0:(ldlm_lib.c:2338:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2988.558475] Lustre: 115876:0:(ldlm_lib.c:2338:target_recovery_overseer()) Skipped 2 previous similar messages [ 2988.562132] Lustre: 115876:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2988.568221] Lustre: 115876:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2988.573603] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2988.577128] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2988.585180] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2988.614380] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2988.621683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2990.303144] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2993.634184] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2993.647215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2993.650207] Lustre: Skipped 48 previous similar messages [ 3001.172526] Lustre: Failing over lustre-MDT0000 [ 3001.397555] Lustre: server umount lustre-MDT0000 complete [ 3015.240188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.168789] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3020.778137] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3020.781937] Lustre: Skipped 4 previous similar messages [ 3020.801045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 3020.801102] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 3022.485054] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3023.246278] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3027.468088] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 07:24:12 (1756812252) [ 3027.512793] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnecting [ 3027.516161] Lustre: Skipped 2 previous similar messages [ 3031.426228] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 07:24:16 (1756812256) [ 3031.851330] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3031.853433] LustreError: 6507:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff988329d83100 x1842148073482496/t0(0) o700->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:504/0 lens 264/248 e 0 to 0 dl 1756812269 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 3048.938206] Lustre: Failing over lustre-MDT0000 [ 3049.173996] Lustre: server umount lustre-MDT0000 complete [ 3063.124235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3065.393387] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3067.908498] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3067.913547] Lustre: Skipped 4 previous similar messages [ 3068.424974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 3068.425054] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 3070.122958] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3070.900272] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3076.095738] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 07:25:01 (1756812301) [ 3077.373443] Lustre: Failing over lustre-OST0000 [ 3077.457041] Lustre: server umount lustre-OST0000 complete [ 3091.405071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3093.974719] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3097.725559] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3098.445746] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3165.106297] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 07:26:30 (1756812390) [ 3168.611043] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3169.793400] Lustre: Failing over lustre-MDT0000 [ 3170.087888] Lustre: server umount lustre-MDT0000 complete [ 3184.955964] LDISKFS-fs (dm-0): recovery complete [ 3184.958072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.599199] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3190.335272] Lustre: *** cfs_fail_loc=216, val=0*** [ 3190.335342] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 3190.337547] LustreError: 121650:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 3191.392569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 3253.666183] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 07:27:59 (1756812479) [ 3262.246726] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 07:28:07 (1756812487) [ 3263.459245] Lustre: Failing over lustre-MDT0000 [ 3263.687328] Lustre: server umount lustre-MDT0000 complete [ 3277.180054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.281558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 3277.285628] Lustre: Skipped 6 previous similar messages [ 3277.360723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3277.364064] Lustre: Skipped 8 previous similar messages [ 3279.028645] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3282.408610] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 3282.410475] LustreError: 123214:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff9882072b9500 x1842148073602560/t0(0) o101->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:39/0 lens 328/344 e 0 to 0 dl 1756812559 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 3337.730712] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnected, waiting for 2 clients in recovery for 1:17 [ 3337.737279] Lustre: 123214:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3337.760381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 3337.760402] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 3339.351255] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3339.993260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3344.159951] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 07:29:29 (1756812569) [ 3345.588343] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3348.947063] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3349.747232] Lustre: Failing over lustre-MDT0000 [ 3349.934995] Lustre: server umount lustre-MDT0000 complete [ 3365.189116] LDISKFS-fs (dm-0): recovery complete [ 3365.190791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.934207] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3370.492413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3137) [ 3370.492422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3105) [ 3371.817496] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3372.470695] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3376.437743] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 07:30:01 (1756812601) [ 3376.914366] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3381.125462] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3381.864174] Lustre: Failing over lustre-MDT0000 [ 3381.966188] Lustre: server umount lustre-MDT0000 complete [ 3396.882146] LDISKFS-fs (dm-0): recovery complete [ 3396.884429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.772646] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3402.236680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 3402.236680] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3137) [ 3403.930274] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3404.718528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3409.207138] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 07:30:34 (1756812634) [ 3409.727843] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3414.201301] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3415.029342] Lustre: Failing over lustre-MDT0000 [ 3415.134773] Lustre: server umount lustre-MDT0000 complete [ 3431.128716] LDISKFS-fs (dm-0): recovery complete [ 3431.132473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.254970] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3436.553174] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 3436.554257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3201) [ 3442.183732] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 07:31:07 (1756812667) [ 3443.804727] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3443.806819] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 3443.809212] LustreError: 6501:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff9883338f5850 x1842148073655040/t257698037777(0) o35->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:161/0 lens 392/456 e 0 to 0 dl 1756812681 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3445.214117] Lustre: Failing over lustre-MDT0000 [ 3445.401618] Lustre: server umount lustre-MDT0000 complete [ 3459.551942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3461.551197] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3465.222285] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 3465.222390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3233) [ 3465.237215] Lustre: 6501:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff98832f889180 x1842148073655040/t257698037777(0) o35->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:182/0 lens 392/456 e 0 to 0 dl 1756812702 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3467.092791] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3467.942640] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3472.822419] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 07:31:38 (1756812698) [ 3473.446197] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3473.449226] LustreError: 8408:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff9882072ba300 x1842148073668480/t261993005072(0) o36->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:234/0 lens 504/448 e 0 to 0 dl 1756812754 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3478.443241] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3479.355967] Lustre: Failing over lustre-MDT0000 [ 3479.449794] Lustre: server umount lustre-MDT0000 complete [ 3494.689608] LDISKFS-fs (dm-0): recovery complete [ 3494.691790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.574928] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3500.023342] Lustre: 6497:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff988329d74e00 x1842148073668480/t261993005072(0) o36->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:261/0 lens 504/2880 e 0 to 0 dl 1756812781 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3500.028257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 3500.028330] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233) [ 3501.665154] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3502.402443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3506.626640] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 07:32:12 (1756812732) [ 3507.143674] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3507.145283] LustreError: 6497:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff98833d47ce00 x1842148073682048/t266287972368(0) o36->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:268/0 lens 504/448 e 0 to 0 dl 1756812788 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3508.537320] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3511.801860] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3512.677707] Lustre: Failing over lustre-MDT0000 [ 3512.789664] Lustre: server umount lustre-MDT0000 complete [ 3528.422017] LDISKFS-fs (dm-0): recovery complete [ 3528.424632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3530.300793] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3533.821061] Lustre: 6498:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff98833b161f80 x1842148073682048/t266287972368(0) o36->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:295/0 lens 504/2880 e 0 to 0 dl 1756812815 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3533.827874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 3533.828090] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3265) [ 3533.833741] Lustre: 6498:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 3539.230746] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 07:32:44 (1756812764) [ 3539.714669] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3539.719140] Lustre: Skipped 1 previous similar message [ 3539.721178] LustreError: 6498:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff98833d80dc00 x1842148073695616/t270582939664(0) o36->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:300/0 lens 504/448 e 0 to 0 dl 1756812820 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3539.731140] LustreError: 6498:0:(ldlm_lib.c:3277:target_send_reply_msg()) Skipped 1 previous similar message [ 3541.117792] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3541.119703] Lustre: Skipped 1 previous similar message [ 3545.248385] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3546.276266] Lustre: Failing over lustre-MDT0000 [ 3546.381732] Lustre: server umount lustre-MDT0000 complete [ 3561.779689] LDISKFS-fs (dm-0): recovery complete [ 3561.782787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.660781] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3567.089929] Lustre: 18188:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff988206ad5c00 x1842148073695616/t270582939664(0) o36->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:328/0 lens 504/2880 e 0 to 0 dl 1756812848 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3567.097402] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3297) [ 3567.097402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 3571.809115] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 07:33:17 (1756812797) [ 3572.274529] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3573.640425] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3573.642711] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 3573.644664] LustreError: 6501:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff988329f9dc00 x1842148073708928/t274877906960(0) o35->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:290/0 lens 392/456 e 0 to 0 dl 1756812810 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3577.846605] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3578.701155] Lustre: Failing over lustre-MDT0000 [ 3578.836446] Lustre: server umount lustre-MDT0000 complete [ 3582.432751] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.432826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3582.433119] LustreError: 6503:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3582.433126] LustreError: 6503:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 181 previous similar messages [ 3582.438194] Lustre: Skipped 52 previous similar messages [ 3582.441240] LustreError: Skipped 12 previous similar messages [ 3594.044371] LDISKFS-fs (dm-0): recovery complete [ 3594.047169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.100357] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3594.106138] LustreError: Skipped 11 previous similar messages [ 3594.181189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3594.183821] Lustre: Skipped 12 previous similar messages [ 3595.831522] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3599.333175] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3599.335589] Lustre: Skipped 50 previous similar messages [ 3599.359449] Lustre: 6501:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff98833a937800 x1842148073708928/t274877906960(0) o35->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:316/0 lens 392/456 e 0 to 0 dl 1756812836 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3599.362821] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 3599.362853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3361) [ 3604.462619] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 07:33:49 (1756812829) [ 3604.839628] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 3664.400309] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnecting [ 3664.403459] Lustre: Skipped 4 previous similar messages [ 3664.406646] Lustre: 8408:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff988306334700 x1842148073719936/t279172874255(0) o101->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:425/0 lens 664/3488 e 0 to 0 dl 1756812945 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 3666.766413] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 07:34:52 (1756812892) [ 3669.290210] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3669.940216] Lustre: Failing over lustre-MDT0000 [ 3670.002178] LustreError: 139150:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3670.005347] LustreError: 139150:0:(obd_class.h:479:obd_check_dev()) Skipped 107 previous similar messages [ 3670.028792] Lustre: server umount lustre-MDT0000 complete [ 3684.363986] LDISKFS-fs (dm-0): recovery complete [ 3684.365748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.866908] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3684.871135] Lustre: Skipped 11 previous similar messages [ 3685.810598] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3689.966927] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3689.970596] Lustre: Skipped 12 previous similar messages [ 3689.985959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3393) [ 3689.985995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 3691.159212] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3691.684197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3705.260809] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 07:35:30 (1756812930) [ 3708.006899] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3708.591269] Lustre: Failing over lustre-MDT0000 [ 3708.777838] Lustre: server umount lustre-MDT0000 complete [ 3723.047644] LDISKFS-fs (dm-0): recovery complete [ 3723.050304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.670895] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3728.398264] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3393) [ 3728.398264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 3729.756171] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3730.411328] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3733.017152] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3738.780649] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 07:36:04 (1756812964) [ 3750.875141] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3751.630033] Lustre: Failing over lustre-MDT0000 [ 3751.956184] Lustre: server umount lustre-MDT0000 complete [ 3766.345784] LDISKFS-fs (dm-0): recovery complete [ 3766.347283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3767.957192] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3771.977131] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 3771.977131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 3773.188468] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3773.747946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3788.681541] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 07:36:53 (1756813013) [ 3792.054632] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3792.775469] Lustre: Failing over lustre-MDT0000 [ 3792.951602] Lustre: server umount lustre-MDT0000 complete [ 3807.198879] LDISKFS-fs (dm-0): recovery complete [ 3807.200715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3808.748959] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3812.871456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 3812.871476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 3814.098201] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3814.626612] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3817.579285] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3818.084482] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3820.141597] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 07:37:25 (1756813045) [ 3826.023482] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3887.442688] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3887.444285] Lustre: Skipped 1 previous similar message [ 3887.445738] LustreError: 9281:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff9883081cfc50 x1842148076508544/t296352743435(0) o36->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:648/0 lens 66040/440 e 0 to 0 dl 1756813168 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3887.455394] LustreError: 9281:0:(ldlm_lib.c:3277:target_send_reply_msg()) Skipped 1 previous similar message [ 3947.013179] Lustre: 8408:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9882072bad80 x1842148076508544/t296352743435(0) o36->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:708/0 lens 66040/440 e 0 to 0 dl 1756813228 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3950.310123] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3950.895574] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 07:39:36 (1756813176) [ 3955.315769] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3956.162294] Lustre: Failing over lustre-MDT0000 [ 3956.194215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3956.196316] Lustre: Skipped 3 previous similar messages [ 3956.311157] Lustre: server umount lustre-MDT0000 complete [ 3970.539915] LDISKFS-fs (dm-0): recovery complete [ 3970.541599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3970.670276] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3970.673041] Lustre: Skipped 12 previous similar messages [ 3971.952981] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3975.824711] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 3975.824720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 3976.977530] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3977.558854] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3981.226727] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 07:40:06 (1756813206) [ 3987.043304] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3990.151673] Lustre: Failing over lustre-OST0000 [ 3990.187814] Lustre: server umount lustre-OST0000 complete [ 4004.180478] LDISKFS-fs (dm-2): recovery complete [ 4004.181932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4005.960534] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4017.202880] Lustre: Failing over lustre-OST0000 [ 4017.241949] Lustre: server umount lustre-OST0000 complete [ 4029.939158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4031.857190] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4034.846899] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4035.360990] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4069.535216] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 07:41:35 (1756813295) [ 4070.516206] Lustre: Failing over lustre-MDT0000 [ 4070.707097] Lustre: server umount lustre-MDT0000 complete [ 4083.680788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4085.212161] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4088.827260] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 4088.827261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 4096.678211] Lustre: Failing over lustre-MDT0000 [ 4096.875475] Lustre: server umount lustre-MDT0000 complete [ 4109.583899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4110.971435] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4114.937787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 4114.937804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 4116.185062] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4116.803189] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4120.470781] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 07:42:25 (1756813345) [ 4131.986956] Lustre: Failing over lustre-OST0000 [ 4132.035274] Lustre: server umount lustre-OST0000 complete [ 4144.978267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4146.978921] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4150.147631] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4150.760198] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4154.730969] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 07:43:00 (1756813380) [ 4155.516529] Lustre: Failing over lustre-MDT0000 [ 4155.888490] Lustre: server umount lustre-MDT0000 complete [ 4159.203586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.240058] Lustre: *** cfs_fail_loc=605, val=0*** [ 4159.241491] LustreError: 156901:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc0a4bb70 failed: rc = -95 [ 4159.244410] LustreError: 156901:0:(obd_config.c:782:class_setup()) setup MGS failed (-95) [ 4159.246272] LustreError: 156901:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 4159.248385] LustreError: 156901:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 4159.250359] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 4159.252084] LustreError: 156901:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-MDT0000 [ 4159.256924] Lustre: server umount lustre-MDT0000 complete [ 4159.258809] LustreError: 156901:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 4162.104995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.615150] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4166.420407] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 07:43:12 (1756813392) [ 4167.670880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 4167.670993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 4170.281552] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4171.357183] Lustre: Failing over lustre-MDT0000 [ 4171.663731] Lustre: server umount lustre-MDT0000 complete [ 4182.530717] LustreError: 8408:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.537233] LustreError: 8408:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 151 previous similar messages [ 4185.919627] LDISKFS-fs (dm-0): recovery complete [ 4185.921335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.388874] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4187.652270] Lustre: *** cfs_fail_loc=707, val=0*** [ 4247.051416] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 4247.065929] Lustre: 159054:0:(ldlm_lib.c:2020:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4247.069588] Lustre: 159054:0:(ldlm_lib.c:2020:extend_recovery_timer()) Skipped 1 previous similar message [ 4247.185228] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4247.187896] Lustre: Skipped 44 previous similar messages [ 4247.203050] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 4247.203050] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 4248.636210] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4249.267914] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4253.426403] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 07:44:38 (1756813478) [ 4277.212454] LustreError: 6497:0:(service.c:2535:ptlrpc_server_handle_request()) @@@ HIT req@ffff988205f5ea00 x1842148077386624/t0(0) o101->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:239/0 lens 664/0 e 0 to 0 dl 1756813514 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4277.220881] LustreError: 6497:0:(service.c:2536:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 4288.248135] LustreError: 6497:0:(service.c:2536:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4288.254404] LustreError: 38238:0:(service.c:2535:ptlrpc_server_handle_request()) @@@ HIT req@ffff9883081e6300 x1842148077387648/t0(0) o35->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:250/0 lens 392/0 e 0 to 0 dl 1756813525 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4293.603717] LustreError: 8383:0:(service.c:2535:ptlrpc_server_handle_request()) @@@ HIT req@ffff988306336680 x1842148083096704/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:255/0 lens 544/0 e 0 to 0 dl 1756813530 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 4293.610187] LustreError: 8383:0:(service.c:2535:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 4300.156640] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 07:45:25 (1756813525) [ 4323.782041] LustreError: 41008:0:(tgt_handler.c:2785:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 4334.808058] LustreError: 41008:0:(tgt_handler.c:2785:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 4337.994863] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 07:46:03 (1756813563) [ 4360.949939] LustreError: 6497:0:(service.c:2535:ptlrpc_server_handle_request()) @@@ HIT req@ffff988333389880 x1842148077409664/t0(0) o101->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:323/0 lens 576/0 e 0 to 0 dl 1756813598 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4360.958688] LustreError: 6497:0:(service.c:2536:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 4366.056089] LustreError: 6497:0:(service.c:2536:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4366.062221] LustreError: 6497:0:(service.c:2535:ptlrpc_server_handle_request()) @@@ HIT req@ffff98832c899500 x1842148077410432/t0(0) o101->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:328/0 lens 664/0 e 0 to 0 dl 1756813603 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4376.848117] LustreError: 6497:0:(service.c:2536:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4376.854695] LustreError: 6497:0:(service.c:2535:ptlrpc_server_handle_request()) @@@ HIT req@ffff98832cf0ad80 x1842148077431808/t0(0) o101->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:379/0 lens 664/0 e 0 to 0 dl 1756813654 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4376.860510] LustreError: 6497:0:(service.c:2535:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 4388.852486] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 07:46:54 (1756813614) [ 4484.019402] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 07:48:29 (1756813709) [ 4507.050732] LustreError: 18188:0:(service.c:2535:ptlrpc_server_handle_request()) @@@ HIT req@ffff988329d74a80 x1842148077488000/t0(0) o101->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:469/0 lens 576/0 e 0 to 0 dl 1756813744 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4507.058796] LustreError: 18188:0:(service.c:2535:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 4507.061616] LustreError: 18188:0:(service.c:2536:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 4507.064528] LustreError: 18188:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 4507.480107] LustreError: 18188:0:(service.c:2536:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4523.306159] LustreError: 18188:0:(service.c:2536:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 4523.308861] LustreError: 18188:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 4523.728123] LustreError: 18188:0:(service.c:2536:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4523.730472] LustreError: 18188:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 4539.138410] LustreError: 163255:0:(service.c:2535:ptlrpc_server_handle_request()) @@@ HIT req@ffff98832f889f80 x1842148077507328/t0(0) o36->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:501/0 lens 504/0 e 0 to 0 dl 1756813776 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4539.144470] LustreError: 163255:0:(service.c:2535:ptlrpc_server_handle_request()) Skipped 82 previous similar messages [ 4551.230104] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 07:49:36 (1756813776) [ 4575.786831] Lustre: DEBUG MARKER: phase 2 [ 4578.984717] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 07:50:04 (1756813804) [ 4649.535306] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 07:51:15 (1756813875) [ 4650.009085] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 4650.525605] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 07:51:16 (1756813876) [ 4652.288747] Lustre: DEBUG MARKER: Started rundbench load pid=133640 ... [ 4656.045412] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4657.597457] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 4658.279805] Lustre: Failing over lustre-MDT0000 [ 4658.295665] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 4658.494883] LustreError: 165957:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4658.496714] LustreError: 165957:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages [ 4658.523739] Lustre: server umount lustre-MDT0000 complete [ 4662.240567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4662.241540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4662.244113] Lustre: Skipped 47 previous similar messages [ 4662.246510] LustreError: Skipped 10 previous similar messages [ 4672.544785] LDISKFS-fs (dm-0): recovery complete [ 4672.546191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4672.580248] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.584846] LustreError: Skipped 9 previous similar messages [ 4672.652791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4672.654712] Lustre: Skipped 12 previous similar messages [ 4672.664723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4672.668457] Lustre: Skipped 7 previous similar messages [ 4673.979374] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4674.051135] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4674.054634] Lustre: Skipped 11 previous similar messages [ 4678.391032] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4678.393471] Lustre: Skipped 11 previous similar messages [ 4678.410717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5120 to 0x280000401:5153) [ 4678.410894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5085 to 0x2c0000401:5121) [ 4679.796084] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4680.430497] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4685.803230] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4687.410649] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 4688.159969] Lustre: Failing over lustre-MDT0001 [ 4688.352856] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4688.439739] Lustre: server umount lustre-MDT0001 complete [ 4703.044357] LDISKFS-fs (dm-1): recovery complete [ 4703.046499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4704.625214] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4708.342077] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:422 to 0x280000400:449) [ 4708.342084] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:449) [ 4709.717560] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4710.326375] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4715.682423] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4717.277219] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 4718.009231] Lustre: Failing over lustre-MDT0000 [ 4718.284837] Lustre: server umount lustre-MDT0000 complete [ 4732.573506] LDISKFS-fs (dm-0): recovery complete [ 4732.574837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.116611] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4738.437288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5461 to 0x2c0000401:5505) [ 4738.437364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5494 to 0x280000401:5537) [ 4739.674269] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4740.264774] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4745.261477] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4746.794232] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 4747.481705] Lustre: Failing over lustre-MDT0001 [ 4747.669584] Lustre: server umount lustre-MDT0001 complete [ 4762.106202] LDISKFS-fs (dm-1): recovery complete [ 4762.108249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4763.610604] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4767.733861] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:481) [ 4767.733909] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:422 to 0x280000400:481) [ 4768.947915] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4769.550401] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4798.001632] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 07:53:43 (1756814023) [ 4921.127068] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4931.929140] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 4932.588459] Lustre: Failing over lustre-MDT0000 [ 4932.652792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4932.655377] Lustre: Skipped 2 previous similar messages [ 4932.850028] Lustre: server umount lustre-MDT0000 complete [ 4935.687133] LustreError: 6498:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4935.694659] LustreError: 6498:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 55 previous similar messages [ 4946.942170] LDISKFS-fs (dm-0): recovery complete [ 4946.943552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4948.300269] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4952.546881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4952.550474] Lustre: Skipped 14 previous similar messages [ 4954.726398] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7406 to 0x2c0000401:7425) [ 4954.729130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7437 to 0x280000401:7457) [ 4956.044888] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4956.619070] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5080.911481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5091.722914] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 5092.382155] Lustre: Failing over lustre-MDT0000 [ 5092.409120] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 5096.962969] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 5096.966014] Lustre: Skipped 5 previous similar messages [ 5098.080188] Lustre: server umount lustre-MDT0000 complete [ 5112.515253] LDISKFS-fs (dm-0): recovery complete [ 5112.516852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5114.024365] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5120.106553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8828 to 0x280000401:8865) [ 5120.109476] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8797 to 0x2c0000401:8833) [ 5121.326659] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5121.866981] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5141.045955] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 07:59:26 (1756814366) [ 5263.972737] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5274.749195] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 5275.467504] Lustre: Failing over lustre-MDT0000 [ 5275.482853] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 5276.640959] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.644276] Lustre: Skipped 21 previous similar messages [ 5280.794587] LustreError: 176614:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5280.796661] LustreError: 176614:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 5280.847983] Lustre: server umount lustre-MDT0000 complete [ 5294.971267] LDISKFS-fs (dm-0): recovery complete [ 5294.973628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5295.012328] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5295.018304] LustreError: Skipped 3 previous similar messages [ 5295.090802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5295.094688] Lustre: Skipped 5 previous similar messages [ 5295.106422] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5295.108576] Lustre: Skipped 5 previous similar messages [ 5296.445174] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5296.642977] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5296.646108] Lustre: Skipped 5 previous similar messages [ 5301.242965] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5301.245446] Lustre: Skipped 5 previous similar messages [ 5301.259300] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9353 to 0x2c0000401:9377) [ 5301.259312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9385 to 0x280000401:9409) [ 5302.503077] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5303.079430] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5427.260622] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5438.080646] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 5438.738041] Lustre: Failing over lustre-MDT0000 [ 5438.751612] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -19 [ 5438.753790] LustreError: Skipped 5 previous similar messages [ 5438.757727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5438.759240] Lustre: Skipped 5 previous similar messages [ 5445.209941] Lustre: server umount lustre-MDT0000 complete [ 5459.372359] LDISKFS-fs (dm-0): recovery complete [ 5459.373991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5460.716174] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5465.637960] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9793 to 0x2c0000401:9825) [ 5465.639069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9824 to 0x280000401:9857) [ 5466.778225] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5467.327473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5470.572980] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 08:04:56 (1756814696) [ 5593.917824] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5604.600888] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 5605.191118] Lustre: Failing over lustre-MDT0000 [ 5605.384741] Lustre: server umount lustre-MDT0000 complete [ 5607.904916] LustreError: 9281:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.909216] LustreError: 9281:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 61 previous similar messages [ 5619.391341] LDISKFS-fs (dm-0): recovery complete [ 5619.392700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5620.788681] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5624.803969] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5624.807561] Lustre: Skipped 15 previous similar messages [ 5628.155447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11886 to 0x280000401:11905) [ 5628.155460] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11853 to 0x2c0000401:11873) [ 5629.359017] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5629.928942] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5754.080834] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5764.833185] Lustre: DEBUG MARKER: test_70e fail mds2 2 times [ 5765.479275] Lustre: Failing over lustre-MDT0001 [ 5765.488395] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5765.491425] Lustre: Skipped 6 previous similar messages [ 5765.727482] Lustre: server umount lustre-MDT0001 complete [ 5779.662457] LDISKFS-fs (dm-1): recovery complete [ 5779.663836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5781.040339] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5785.083624] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3999 to 0x280000400:4033) [ 5785.083655] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4000 to 0x2c0000400:4033) [ 5786.347821] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5786.945411] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5790.762218] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 08:10:16 (1756815016) [ 5796.665483] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5798.204722] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5798.860625] Lustre: Failing over lustre-OST0000 [ 5798.904481] Lustre: server umount lustre-OST0000 complete [ 5813.057165] LDISKFS-fs (dm-2): recovery complete [ 5813.058673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5814.983087] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5818.274255] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5818.969265] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5827.851073] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5829.359355] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5829.975465] Lustre: Failing over lustre-OST0000 [ 5830.023431] Lustre: server umount lustre-OST0000 complete [ 5844.018610] LDISKFS-fs (dm-2): recovery complete [ 5844.019988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5845.943410] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5849.228819] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5849.865334] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5855.591909] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 08:11:21 (1756815081) [ 5978.500449] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5981.090186] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5991.861610] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 5992.495720] Lustre: Failing over lustre-MDT0000 [ 5992.511454] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 5992.622447] LustreError: 188013:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5992.624820] LustreError: 188013:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 5992.669878] Lustre: server umount lustre-MDT0000 complete [ 5992.929704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5992.933802] Lustre: Skipped 17 previous similar messages [ 5993.909686] LustreError: 8399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815220 with bad export cookie 1068264553340000399 [ 5993.910482] Lustre: Failing over lustre-MDT0001 [ 5993.910947] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5993.910953] LustreError: Skipped 2 previous similar messages [ 5993.913117] LustreError: 8399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5994.093169] Lustre: server umount lustre-MDT0001 complete [ 6009.143909] LDISKFS-fs (dm-0): recovery complete [ 6009.146070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6009.173212] LDISKFS-fs (dm-1): recovery complete [ 6009.175452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6014.432162] Lustre: 3635:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756815224/real 1756815224] req@ffff988208abdc00 x1842148125774208/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756815240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6014.441211] Lustre: 3635:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6019.552560] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff988208abed80 x1842148125776384/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6019.673267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.675656] Lustre: Skipped 5 previous similar messages [ 6019.685879] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6019.687989] Lustre: Skipped 5 previous similar messages [ 6021.256085] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6021.384690] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6022.594911] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6022.597280] Lustre: Skipped 5 previous similar messages [ 6024.934690] Lustre: lustre-MDT0001: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6024.937545] Lustre: Skipped 5 previous similar messages [ 6024.955208] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3999 to 0x280000400:4065) [ 6024.955246] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4000 to 0x2c0000400:4065) [ 6036.997391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13882 to 0x280000401:13921) [ 6036.998040] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13850 to 0x2c0000401:13889) [ 6038.396495] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6038.950464] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6039.539562] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6042.891319] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 08:14:28 (1756815268) [ 6045.376471] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6046.254791] Lustre: Failing over lustre-MDT0000 [ 6046.462727] Lustre: server umount lustre-MDT0000 complete [ 6047.200519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6047.202823] LustreError: Skipped 5 previous similar messages [ 6060.447303] LDISKFS-fs (dm-0): recovery complete [ 6060.448646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6061.785169] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6062.596648] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 6077.976420] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 6078.033263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13882 to 0x280000401:13953) [ 6078.033291] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:13921) [ 6079.550193] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6080.052786] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6083.878345] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 08:15:09 (1756815309) [ 6086.430497] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6087.460841] Lustre: Failing over lustre-MDT0000 [ 6087.666826] Lustre: server umount lustre-MDT0000 complete [ 6102.031374] LDISKFS-fs (dm-0): recovery complete [ 6102.033403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6103.423849] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6103.564227] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 6103.567242] LustreError: 189358:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff98820c5d8a80 x1842148166891392/t347892371234(347892371234) o101->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:555/0 lens 520/664 e 0 to 0 dl 1756815340 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6119.954915] Lustre: lustre-MDT0000: Client 4f47911a-29dc-49cd-abfa-4405de2af7b6 (at 192.168.201.6@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 6119.960168] Lustre: 190072:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9882061da680 x1842148166891392/t347892371234(347892371234) o101->4f47911a-29dc-49cd-abfa-4405de2af7b6@192.168.201.6@tcp:572/0 lens 520/3488 e 0 to 0 dl 1756815357 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6119.998261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13955 to 0x280000401:13985) [ 6119.998277] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:13953) [ 6121.482628] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6122.092381] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6125.818653] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 08:15:51 (1756815351) [ 6127.396267] Lustre: Failing over lustre-OST0000 [ 6127.442514] Lustre: server umount lustre-OST0000 complete [ 6128.789364] Lustre: Failing over lustre-MDT0000 [ 6128.872093] Lustre: server umount lustre-MDT0000 complete [ 6142.234084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.710937] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6144.687954] Lustre: lustre-MDT0000: Denying connection for new client a1123ea5-2093-4894-8aa0-deb29b5f2bc0 (at 192.168.201.6@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 6147.592726] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:13985) [ 6152.318966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6153.522618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13955 to 0x280000401:14017) [ 6154.380585] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6158.341168] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 08:16:23 (1756815383) [ 6158.673602] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6158.675053] LustreError: 189333:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff988205e8b480 x1842148127432960/t365072220167(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:610/0 lens 1312/4320 e 0 to 0 dl 1756815395 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6161.079202] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6161.686168] Lustre: Failing over lustre-MDT0000 [ 6162.044961] Lustre: server umount lustre-MDT0000 complete [ 6174.177161] Lustre: 189386:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756815384/real 1756815384] req@ffff988205e88380 x1842148127432960/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1756815400 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6174.187917] Lustre: 189386:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 6176.255394] LDISKFS-fs (dm-0): recovery complete [ 6176.256646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.577128] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6181.882169] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14017) [ 6181.882190] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14049) [ 6182.958215] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6183.463534] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6186.849571] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 08:16:52 (1756815412) [ 6187.217903] LustreError: 190082:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff98820f77d180 x1842148127459968/t369367187468(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:639/0 lens 2520/4320 e 0 to 0 dl 1756815424 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6189.543462] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6192.011893] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6192.671967] Lustre: Failing over lustre-MDT0001 [ 6192.675503] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [ 6197.832161] Lustre: server umount lustre-MDT0001 complete [ 6211.586861] LustreError: 189358:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.591938] LustreError: 189358:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 120 previous similar messages [ 6211.850762] LDISKFS-fs (dm-1): recovery complete [ 6211.852096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6213.114734] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6217.197134] Lustre: 194068:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff98820f77e300 x1842148174035584/t21474838943(0) o36->a1123ea5-2093-4894-8aa0-deb29b5f2bc0@192.168.201.6@tcp:693/0 lens 560/2880 e 0 to 0 dl 1756815478 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6217.204292] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4076 to 0x2c0000400:4097) [ 6217.204317] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4076 to 0x280000400:4097) [ 6218.350950] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6218.887668] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6222.484950] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 08:17:28 (1756815448) [ 6222.819408] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6222.821065] Lustre: Skipped 1 previous similar message [ 6222.822202] LustreError: 189333:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff98833af95500 x1842148127492736/t369367187489(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:675/0 lens 2520/4320 e 0 to 0 dl 1756815460 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6225.230788] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6227.602682] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6228.197060] Lustre: Failing over lustre-MDT0000 [ 6228.555597] Lustre: server umount lustre-MDT0000 complete [ 6242.538019] LDISKFS-fs (dm-0): recovery complete [ 6242.539607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6243.879596] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6247.907008] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6247.909019] Lustre: Skipped 36 previous similar messages [ 6247.952459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14081) [ 6247.952459] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14049) [ 6249.036248] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6249.541142] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6251.452788] Lustre: Failing over lustre-MDT0001 [ 6251.551878] Lustre: server umount lustre-MDT0001 complete [ 6265.594441] LDISKFS-fs (dm-1): recovery complete [ 6265.595793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6266.958167] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6270.966632] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4108 to 0x2c0000400:4129) [ 6270.966644] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4108 to 0x280000400:4129) [ 6272.021859] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6272.576369] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6276.120846] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 08:18:21 (1756815501) [ 6281.782126] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6284.570947] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6285.270330] Lustre: Failing over lustre-MDT0000 [ 6285.548942] Lustre: server umount lustre-MDT0000 complete [ 6286.794578] LustreError: 8399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815513 with bad export cookie 1068264553342059568 [ 6286.796187] Lustre: Failing over lustre-MDT0001 [ 6286.797963] LustreError: 8399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6292.960915] LustreError: 204993:0:(client.c:1371:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff988206c56680 x1842148127552768/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6292.973804] LustreError: 204993:0:(client.c:1371:ptlrpc_import_delay_req()) Skipped 1 previous similar message [ 6292.978942] LustreError: 204993:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6293.116446] Lustre: server umount lustre-MDT0001 complete [ 6307.720174] LDISKFS-fs (dm-1): recovery complete [ 6307.720212] LDISKFS-fs (dm-0): recovery complete [ 6307.722397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6307.723632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6307.798130] LustreError: 206090:0:(llog.c:1602:llog_backup()) MGC192.168.201.106@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6307.800923] Lustre: 206090:0:(mgc_request_server.c:726:mgc_llog_local_copy()) MGC192.168.201.106@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6311.840349] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff988206c56d80 x1842148127554176/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6313.367560] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6313.486991] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6317.589042] Lustre: 206102:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff988206c54a80 x1842148174101888/t30064771140(0) o36->a1123ea5-2093-4894-8aa0-deb29b5f2bc0@192.168.201.6@tcp:39/0 lens 560/2880 e 0 to 0 dl 1756815579 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6317.600610] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4140 to 0x280000400:4161) [ 6317.600775] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4140 to 0x2c0000400:4161) [ 6317.612406] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14081) [ 6317.612405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14113) [ 6318.839877] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6319.362349] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6319.870916] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6323.314439] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 08:19:08 (1756815548) [ 6323.660914] LustreError: 206848:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff988320e33480 x1842148174134144/t34359738436(0) o36->a1123ea5-2093-4894-8aa0-deb29b5f2bc0@192.168.201.6@tcp:59/0 lens 560/448 e 0 to 0 dl 1756815599 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6323.669904] LustreError: 206848:0:(ldlm_lib.c:3277:target_send_reply_msg()) Skipped 1 previous similar message [ 6329.086317] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6329.740368] Lustre: Failing over lustre-MDT0000 [ 6330.139032] Lustre: server umount lustre-MDT0000 complete [ 6344.054896] LDISKFS-fs (dm-0): recovery complete [ 6344.056985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6345.415275] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6349.300866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14145) [ 6349.300878] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14113) [ 6350.419461] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6350.940729] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6377.987560] Lustre: lustre-MDT0001: Client a1123ea5-2093-4894-8aa0-deb29b5f2bc0 (at 192.168.201.6@tcp) reconnecting [ 6377.992132] Lustre: Skipped 2 previous similar messages [ 6377.996493] Lustre: 206103:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff988332e2ad80 x1842148174134144/t34359738436(0) o36->a1123ea5-2093-4894-8aa0-deb29b5f2bc0@192.168.201.6@tcp:114/0 lens 560/2880 e 0 to 0 dl 1756815654 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6380.857479] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 08:20:06 (1756815606) [ 6381.256595] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6381.259497] Lustre: Skipped 2 previous similar messages [ 6383.985208] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6384.602573] Lustre: Failing over lustre-MDT0001 [ 6384.692765] Lustre: server umount lustre-MDT0001 complete [ 6398.694331] LDISKFS-fs (dm-1): recovery complete [ 6398.695950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6400.070528] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6404.085350] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4182 to 0x280000400:4225) [ 6404.085382] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4182 to 0x2c0000400:4225) [ 6405.208511] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6405.788356] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6409.431862] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 08:20:35 (1756815635) [ 6415.162128] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6417.532084] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6418.126067] Lustre: Failing over lustre-MDT0000 [ 6418.343911] Lustre: server umount lustre-MDT0000 complete [ 6425.103743] Lustre: 206102:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff98833af97b80 x1842148174197504/t38654705732(0) o36->a1123ea5-2093-4894-8aa0-deb29b5f2bc0@192.168.201.6@tcp:122/0 lens 560/2880 e 0 to 0 dl 1756815662 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6425.110037] Lustre: 206102:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6432.414089] LDISKFS-fs (dm-0): recovery complete [ 6432.415479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.849888] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6437.890524] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14145) [ 6437.890576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14177) [ 6438.938299] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6439.443996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6441.321373] Lustre: Failing over lustre-MDT0001 [ 6441.407618] Lustre: server umount lustre-MDT0001 complete [ 6455.424539] LDISKFS-fs (dm-1): recovery complete [ 6455.426770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6456.758222] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6460.916438] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4236 to 0x280000400:4257) [ 6460.916476] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4236 to 0x2c0000400:4257) [ 6462.019226] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6462.603239] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6466.178757] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 08:21:31 (1756815691) [ 6466.529606] LustreError: 206101:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff98833d441500 x1842148174233856/t42949673028(0) o36->a1123ea5-2093-4894-8aa0-deb29b5f2bc0@192.168.201.6@tcp:163/0 lens 560/448 e 0 to 0 dl 1756815703 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6466.536761] LustreError: 206101:0:(ldlm_lib.c:3277:target_send_reply_msg()) Skipped 2 previous similar messages [ 6472.028989] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6474.591530] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6475.250807] Lustre: Failing over lustre-MDT0000 [ 6475.630269] Lustre: server umount lustre-MDT0000 complete [ 6476.826766] LustreError: 8399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815703 with bad export cookie 1068264553342073246 [ 6476.827895] Lustre: Failing over lustre-MDT0001 [ 6476.832429] LustreError: 8399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6476.985543] Lustre: server umount lustre-MDT0001 complete [ 6491.842795] LDISKFS-fs (dm-0): recovery complete [ 6491.844215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6491.856897] LDISKFS-fs (dm-1): recovery complete [ 6491.858588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6497.760304] Lustre: 3636:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756815707/real 1756815707] req@ffff9883385a2d80 x1842148127716224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756815723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6497.772599] Lustre: 3636:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6501.856456] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9883385a2d80 x1842148127718144/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6503.323064] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6503.451122] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6508.495472] Lustre: 216694:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff988341f20380 x1842148174233856/t42949673028(0) o36->a1123ea5-2093-4894-8aa0-deb29b5f2bc0@192.168.201.6@tcp:205/0 lens 560/2880 e 0 to 0 dl 1756815745 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6508.503050] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4268 to 0x280000400:4289) [ 6508.503074] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4268 to 0x2c0000400:4289) [ 6516.701905] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14177) [ 6516.701940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14209) [ 6518.169202] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6518.705401] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6519.207805] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6522.836286] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 08:22:28 (1756815748) [ 6525.478726] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6526.141738] Lustre: Failing over lustre-MDT0001 [ 6526.146232] LustreError: 218371:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200029c11:0x17:0x0].0x0 (ffff9883025c3a00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6526.155919] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [ 6526.157665] Lustre: Skipped 8 previous similar messages [ 6532.131836] Lustre: server umount lustre-MDT0001 complete [ 6546.378821] LDISKFS-fs (dm-1): recovery complete [ 6546.380272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6547.812670] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6551.539727] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4321) [ 6551.539753] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4300 to 0x2c0000400:4321) [ 6552.767227] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6553.288357] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6556.606053] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 08:23:02 (1756815782) [ 6559.587565] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6560.351823] Lustre: Failing over lustre-MDT0000 [ 6560.733809] Lustre: server umount lustre-MDT0000 complete [ 6575.166809] LDISKFS-fs (dm-0): recovery complete [ 6575.168110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6576.694678] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6580.734926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14241) [ 6580.734939] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14209) [ 6581.949572] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6582.497785] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6585.996230] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 08:23:31 (1756815811) [ 6588.939148] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6591.460260] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6592.058018] Lustre: Failing over lustre-MDT0000 [ 6592.420845] Lustre: server umount lustre-MDT0000 complete [ 6596.064731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6596.070181] Lustre: Skipped 66 previous similar messages [ 6606.477764] LDISKFS-fs (dm-0): recovery complete [ 6606.480347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6606.518144] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.521220] LustreError: Skipped 10 previous similar messages [ 6607.897982] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6611.965953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14241) [ 6611.965963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14273) [ 6613.064682] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6613.620880] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6615.615118] Lustre: Failing over lustre-MDT0001 [ 6615.685982] LustreError: 223712:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6615.688177] LustreError: 223712:0:(obd_class.h:479:obd_check_dev()) Skipped 163 previous similar messages [ 6615.709732] Lustre: server umount lustre-MDT0001 complete [ 6629.791715] LDISKFS-fs (dm-1): recovery complete [ 6629.793131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6629.902568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6629.904974] Lustre: Skipped 20 previous similar messages [ 6629.913898] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6629.917428] Lustre: Skipped 20 previous similar messages [ 6631.055092] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6631.057281] Lustre: Skipped 20 previous similar messages [ 6631.188685] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6634.981217] Lustre: lustre-MDT0001: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6634.983899] Lustre: Skipped 20 previous similar messages [ 6634.997636] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4353) [ 6634.997660] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4300 to 0x2c0000400:4353) [ 6636.107841] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6636.691600] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6640.069400] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 08:24:25 (1756815865) [ 6640.437527] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6640.439872] Lustre: Skipped 5 previous similar messages [ 6642.800106] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6645.231378] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6645.854776] Lustre: Failing over lustre-MDT0000 [ 6646.064084] Lustre: server umount lustre-MDT0000 complete [ 6647.211808] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756815873 with bad export cookie 1068264553342083613 [ 6647.213221] Lustre: Failing over lustre-MDT0001 [ 6647.215488] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6652.896915] LustreError: 226155:0:(client.c:1371:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff98830810a680 x1842148127857152/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6652.905539] LustreError: 226155:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6653.059628] Lustre: server umount lustre-MDT0001 complete [ 6667.910629] LDISKFS-fs (dm-0): recovery complete [ 6667.912877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6667.917766] LDISKFS-fs (dm-1): recovery complete [ 6667.920208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6671.840650] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff98833d337b80 x1842148127857920/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6672.154795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 6672.157415] LustreError: Skipped 17 previous similar messages [ 6673.602279] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6673.716841] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6677.532764] Lustre: 227269:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9882061da680 x1842148174311808/t55834574850(0) o36->a1123ea5-2093-4894-8aa0-deb29b5f2bc0@192.168.201.6@tcp:399/0 lens 496/2888 e 0 to 0 dl 1756815939 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6677.544636] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4385) [ 6677.544670] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4300 to 0x2c0000400:4385) [ 6677.545711] Lustre: 227269:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6677.583294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14305) [ 6677.583314] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14273) [ 6678.740074] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6679.258426] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6679.736661] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6683.069192] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 08:25:08 (1756815908) [ 6686.032323] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6686.634884] Lustre: Failing over lustre-MDT0000 [ 6686.767685] Lustre: server umount lustre-MDT0000 complete [ 6700.564855] LDISKFS-fs (dm-0): recovery complete [ 6700.566742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6702.023086] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6706.177412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14337) [ 6706.177422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14305) [ 6707.383800] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6707.999118] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6738.435068] Lustre: lustre-MDT0001: Client a1123ea5-2093-4894-8aa0-deb29b5f2bc0 (at 192.168.201.6@tcp) reconnecting [ 6738.439299] Lustre: Skipped 1 previous similar message [ 6740.600578] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 08:26:06 (1756815966) [ 6740.931574] LustreError: 227301:0:(ldlm_lib.c:3277:target_send_reply_msg()) @@@ dropping reply req@ffff98833338b800 x1842148174341376/t60129542149(0) o36->a1123ea5-2093-4894-8aa0-deb29b5f2bc0@192.168.201.6@tcp:438/0 lens 496/456 e 0 to 0 dl 1756815978 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6740.937353] LustreError: 227301:0:(ldlm_lib.c:3277:target_send_reply_msg()) Skipped 5 previous similar messages [ 6743.288948] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6743.924540] Lustre: Failing over lustre-MDT0001 [ 6743.999440] Lustre: server umount lustre-MDT0001 complete [ 6757.806898] LDISKFS-fs (dm-1): recovery complete [ 6757.808968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6759.079086] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6762.997218] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4417) [ 6762.997222] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4300 to 0x2c0000400:4417) [ 6764.134430] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6764.690830] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6768.054912] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 08:26:33 (1756815993) [ 6770.621390] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6772.923830] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6773.496060] Lustre: Failing over lustre-MDT0000 [ 6773.704507] Lustre: server umount lustre-MDT0000 complete [ 6787.379321] LDISKFS-fs (dm-0): recovery complete [ 6787.380586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.577741] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6792.702675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14369) [ 6792.702690] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14337) [ 6794.104565] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6794.783897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6796.763764] Lustre: Failing over lustre-MDT0001 [ 6796.842843] Lustre: server umount lustre-MDT0001 complete [ 6810.674166] LDISKFS-fs (dm-1): recovery complete [ 6810.675577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6812.024850] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6816.241384] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4300 to 0x2c0000400:4449) [ 6816.241399] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4449) [ 6817.299270] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6817.797510] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6821.280676] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 08:27:26 (1756816046) [ 6824.112538] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6826.520548] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6827.114105] Lustre: Failing over lustre-MDT0000 [ 6827.380077] Lustre: server umount lustre-MDT0000 complete [ 6828.530046] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756816054 with bad export cookie 1068264553342091103 [ 6828.531532] Lustre: Failing over lustre-MDT0001 [ 6828.535836] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6828.548069] LustreError: 233229:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6828.553729] LustreError: 233229:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 202 previous similar messages [ 6828.659987] Lustre: server umount lustre-MDT0001 complete [ 6843.201605] LDISKFS-fs (dm-0): recovery complete [ 6843.203953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.208321] LDISKFS-fs (dm-1): recovery complete [ 6843.211961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6846.944157] Lustre: 3636:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756816057/real 1756816057] req@ffff9883143c0a80 x1842148127997824/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756816073 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6846.950395] Lustre: 3636:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [ 6853.088351] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9883143c1880 x1842148128000512/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6854.687495] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6854.794845] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6857.441995] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6857.443842] Lustre: Skipped 66 previous similar messages [ 6868.500034] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4300 to 0x280000400:4481) [ 6868.500197] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4300 to 0x2c0000400:4481) [ 6868.512261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14401) [ 6868.514148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13891 to 0x2c0000401:14369) [ 6869.835983] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6870.367320] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6870.883018] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6874.326426] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 08:28:19 (1756816099) [ 6874.882819] Lustre: 239138:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a1123ea5-2093-4894-8aa0-deb29b5f2bc0 at adminstrative request [ 6879.369982] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 08:28:25 (1756816105) [ 6880.990734] Lustre: Failing over lustre-MDT0000 [ 6881.172627] Lustre: server umount lustre-MDT0000 complete [ 6893.824450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6895.208842] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6899.216472] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14421 to 0x2c0000401:14465) [ 6899.216472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14452 to 0x280000401:14497) [ 6900.399653] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6900.940859] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6904.490697] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 08:28:50 (1756816130) [ 6907.994958] Lustre: Failing over lustre-OST0000 [ 6908.051644] Lustre: server umount lustre-OST0000 complete [ 6920.707239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6922.522960] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6925.505701] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6926.028847] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6929.599781] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 08:29:15 (1756816155) [ 6930.905717] Lustre: Failing over lustre-MDT0000 [ 6931.083075] Lustre: server umount lustre-MDT0000 complete [ 6933.324703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6934.593831] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6935.284131] Lustre: lustre-MDT0000: Denying connection for new client d33e229b-7c32-44f1-9135-a50a98e5e106 (at 192.168.201.6@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 6935.292439] Lustre: Skipped 1 previous similar message [ 6938.611039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14452 to 0x280000401:14529) [ 6938.611068] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14421 to 0x2c0000401:14497) [ 6942.932782] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 08:29:28 (1756816168) [ 6945.585713] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6946.585625] Lustre: Failing over lustre-OST0000 [ 6946.620422] Lustre: server umount lustre-OST0000 complete [ 6960.729918] LDISKFS-fs (dm-2): recovery complete [ 6960.732041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6962.593600] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6965.611587] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6966.119314] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6969.524897] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 08:29:55 (1756816195) [ 6972.309750] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6974.170350] Lustre: Failing over lustre-OST0000 [ 6974.203873] Lustre: server umount lustre-OST0000 complete [ 6988.204201] LDISKFS-fs (dm-2): recovery complete [ 6988.205689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6990.048621] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.6@tcp inode [0x200030971:0x5:0x0] object 0x280000401:14531 extent [0-4194303]: client csum cabbad42, server csum 86483677 [ 6990.083799] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6993.123207] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6993.641831] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6997.131693] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 08:30:22 (1756816222) [ 6999.531634] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7001.894441] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7003.782787] Lustre: Failing over lustre-MDT0000 [ 7004.020399] Lustre: server umount lustre-MDT0000 complete [ 7005.135620] LustreError: 144873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756816231 with bad export cookie 1068264553342117185 [ 7005.137919] Lustre: Failing over lustre-OST0000 [ 7005.138584] LustreError: 144873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7005.170397] Lustre: server umount lustre-OST0000 complete [ 7019.084392] LDISKFS-fs (dm-0): recovery complete [ 7019.085752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7030.241163] LustreError: 3632:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff988329f85880 x1842148128145536/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7031.552591] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7035.466764] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14421 to 0x2c0000401:14529) [ 7045.384283] LDISKFS-fs (dm-2): recovery complete [ 7045.385542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7047.027446] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7052.291616] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 08:31:17 (1756816277) [ 7056.995842] Lustre: Failing over lustre-OST0000 [ 7057.038470] Lustre: server umount lustre-OST0000 complete [ 7058.179114] Lustre: Failing over lustre-MDT0000 [ 7058.508392] Lustre: server umount lustre-MDT0000 complete [ 7071.073628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.390104] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7076.344137] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14421 to 0x2c0000401:14561) [ 7082.360247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7084.139482] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7084.810667] Lustre: lustre-OST0000: Denying connection for new client b7a38988-f574-40fd-904e-9c343ac69da6 (at 192.168.201.6@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 7090.178833] Lustre: lustre-OST0000: Denying connection for new client b7a38988-f574-40fd-904e-9c343ac69da6 (at 192.168.201.6@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 7095.298700] Lustre: lustre-OST0000: Denying connection for new client b7a38988-f574-40fd-904e-9c343ac69da6 (at 192.168.201.6@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:58 [ 7100.420071] Lustre: lustre-OST0000: Denying connection for new client b7a38988-f574-40fd-904e-9c343ac69da6 (at 192.168.201.6@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 7110.658730] Lustre: lustre-OST0000: Denying connection for new client b7a38988-f574-40fd-904e-9c343ac69da6 (at 192.168.201.6@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:43 [ 7110.663630] Lustre: Skipped 1 previous similar message [ 7131.138885] Lustre: lustre-OST0000: Denying connection for new client b7a38988-f574-40fd-904e-9c343ac69da6 (at 192.168.201.6@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:22 [ 7131.142632] Lustre: Skipped 3 previous similar messages [ 7154.000111] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 7154.002936] Lustre: 252158:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client d33e229b-7c32-44f1-9135-a50a98e5e106@ [ 7154.008426] Lustre: 252158:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 7154.012198] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 7154.019925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14580 to 0x280000401:14601) [ 7158.210750] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 70 sec [ 7163.152212] Lustre: DEBUG MARKER: free_before: 7646296 free_after: 7646296 [ 7165.266560] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 08:33:10 (1756816390) [ 7166.641254] Lustre: Failing over lustre-OST0001 [ 7166.691554] Lustre: server umount lustre-OST0001 complete [ 7179.684195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7181.596600] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7185.354571] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 08:33:30 (1756816410) [ 7186.516791] Lustre: Failing over lustre-OST0000 [ 7186.562773] Lustre: server umount lustre-OST0000 complete [ 7199.211926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7200.499195] LustreError: 255229:0:(ldlm_lib.c:2836:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 7200.501377] LustreError: 255229:0:(ldlm_lib.c:2836:target_recovery_thread()) Skipped 79 previous similar messages [ 7201.064213] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7206.880154] Lustre: *** cfs_fail_loc=715, val=40*** [ 7216.096629] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [ 7216.101137] Lustre: Skipped 1 previous similar message [ 7222.240196] Lustre: *** cfs_fail_loc=715, val=40*** [ 7222.241367] Lustre: Skipped 2 previous similar messages [ 7232.480779] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 7232.483435] Lustre: Skipped 2 previous similar messages [ 7238.624092] Lustre: *** cfs_fail_loc=715, val=40*** [ 7238.625280] Lustre: Skipped 2 previous similar messages [ 7240.544199] LustreError: 255229:0:(ldlm_lib.c:2836:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7240.546644] LustreError: 255229:0:(ldlm_lib.c:2836:target_recovery_thread()) Skipped 79 previous similar messages [ 7240.551813] Lustre: lustre-OST0000: Recovery over after 0:40, of 3 clients 3 recovered and 0 were evicted. [ 7240.557184] Lustre: Skipped 18 previous similar messages [ 7241.843547] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7242.393172] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7245.839541] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 08:34:31 (1756816471) [ 7246.998866] Lustre: Failing over lustre-MDT0000 [ 7247.190473] LustreError: 256214:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 7247.192230] LustreError: 256214:0:(obd_class.h:479:obd_check_dev()) Skipped 131 previous similar messages [ 7247.215668] Lustre: server umount lustre-MDT0000 complete [ 7247.328709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7247.331676] Lustre: Skipped 54 previous similar messages [ 7259.786240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.819738] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7259.822244] LustreError: Skipped 8 previous similar messages [ 7259.889112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.891472] Lustre: Skipped 19 previous similar messages [ 7259.912110] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7259.913858] Lustre: Skipped 19 previous similar messages [ 7261.007122] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7262.210983] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7262.213309] Lustre: Skipped 19 previous similar messages [ 7265.258840] LustreError: 256667:0:(ldlm_lib.c:2836:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 7268.320155] Lustre: *** cfs_fail_loc=715, val=80*** [ 7268.322487] Lustre: Skipped 2 previous similar messages [ 7278.594798] Lustre: lustre-MDT0000: Client b7a38988-f574-40fd-904e-9c343ac69da6 (at 192.168.201.6@tcp) reconnected, waiting for 2 clients in recovery for 1:27 [ 7278.597624] Lustre: Skipped 1 previous similar message [ 7281.632455] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7284.704100] Lustre: *** cfs_fail_loc=715, val=80*** [ 7284.705390] Lustre: Skipped 1 previous similar message [ 7293.955244] Lustre: lustre-MDT0000: Client b7a38988-f574-40fd-904e-9c343ac69da6 (at 192.168.201.6@tcp) reconnected, waiting for 2 clients in recovery for 1:12 [ 7296.994518] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7300.064257] Lustre: *** cfs_fail_loc=715, val=80*** [ 7300.067252] Lustre: Skipped 1 previous similar message [ 7310.339416] Lustre: lustre-MDT0000: Client b7a38988-f574-40fd-904e-9c343ac69da6 (at 192.168.201.6@tcp) reconnected, waiting for 2 clients in recovery for 0:55 [ 7313.376824] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7316.448185] Lustre: *** cfs_fail_loc=715, val=80*** [ 7316.449560] Lustre: Skipped 1 previous similar message [ 7326.723526] Lustre: lustre-MDT0000: Client b7a38988-f574-40fd-904e-9c343ac69da6 (at 192.168.201.6@tcp) reconnected, waiting for 2 clients in recovery for 0:39 [ 7329.760734] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7345.352158] LustreError: 256667:0:(ldlm_lib.c:2836:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7345.360819] Lustre: 256667:0:(ldlm_lib.c:2882:target_recovery_thread()) too long recovery - read logs [ 7345.366163] LustreError: dumping log to /tmp/lustre-log.1756816571.256667 [ 7345.447617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14615 to 0x280000401:14633) [ 7345.447644] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14574 to 0x2c0000401:14593) [ 7347.057826] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7347.882679] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7352.556193] Lustre: DEBUG MARKER: == replay-single test complete, duration 7178 sec ======== 08:36:17 (1756816577) [ 7353.365477] Lustre: DEBUG MARKER: === replay-single: start cleanup 08:36:18 (1756816578) === [ 7357.438273] Lustre: DEBUG MARKER: === replay-single: finish cleanup 08:36:22 (1756816582) === [ 7358.421119] Lustre: Failing over lustre-MDT0000 [ 7358.494191] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756816584 with bad export cookie 1068264553342130688 [ 7358.495630] LustreError: 3633:0:(client.c:1371:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff98833d641c00 x1842148128371584/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 7358.500379] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7358.509206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7358.512211] Lustre: Skipped 19 previous similar messages [ 7358.571523] Lustre: server umount lustre-MDT0000 complete [ 7375.617605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7377.428743] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7380.998697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14615 to 0x280000401:14665) [ 7380.998819] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14574 to 0x2c0000401:14625) [ 7382.571160] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7383.264625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7391.201976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7391.207956] LustreError: Skipped 18 previous similar messages [ 7393.649992] Lustre: server umount lustre-MDT0000 complete [ 7397.381039] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756816623 with bad export cookie 1068264553342150659 [ 7397.387338] LustreError: 6485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7397.526313] Lustre: server umount lustre-MDT0001 complete [ 7411.711487] Lustre: server umount lustre-OST0000 complete [ 7423.865764] Lustre: server umount lustre-OST0001 complete [ 7429.125469] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 7430.340274] Key type lgssc unregistered [ 7430.457307] LNet: 261237:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7430.459476] LNetError: 261237:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7430.466331] LNet: Removed LNI 192.168.201.106@tcp [ 7430.800105] Key type .llcrypt unregistered [ 7430.801102] Key type ._llcrypt unregistered