[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 645363634 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001025] APIC: Switch to symmetric I/O mode setup [ 0.002081] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003011] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007015] pid_max: default: 32768 minimum: 301 [ 0.008000] LSM: Security Framework initializing [ 0.008000] Yama: becoming mindful. [ 0.008000] SELinux: Initializing. [ 0.008000] *** VALIDATE selinux *** [ 0.015616] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.020251] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024118] *** VALIDATE tmpfs *** [ 0.025518] *** VALIDATE proc *** [ 0.027131] *** VALIDATE cgroup *** [ 0.028007] *** VALIDATE cgroup2 *** [ 0.029274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031126] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033030] Spectre V2 : User space: Vulnerable [ 0.034007] Speculative Store Bypass: Vulnerable [ 0.037592] debug: unmapping init [mem 0xffffffffa1859000-0xffffffffa1860fff] [ 0.038654] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040398] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041023] ... version: 2 [ 0.042009] ... bit width: 48 [ 0.043007] ... generic registers: 4 [ 0.044008] ... value mask: 0000ffffffffffff [ 0.045011] ... max period: 00007fffffffffff [ 0.046013] ... fixed-purpose events: 3 [ 0.047009] ... event mask: 000000070000000f [ 0.048270] rcu: Hierarchical SRCU implementation. [ 0.050385] smp: Bringing up secondary CPUs ... [ 0.051577] x86: Booting SMP configuration: [ 0.052015] .... node #0, CPUs: #1 #2 #3 [ 0.060018] smp: Brought up 1 node, 4 CPUs [ 0.062019] smpboot: Max logical packages: 1 [ 0.063024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.195000] node 0 deferred pages initialised in 130ms [ 0.201463] devtmpfs: initialized [ 0.203637] x86/mm: Memory block size: 128MB [ 0.208585] gcov: version magic: 0x41383552 [ 0.212444] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.216123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.219303] pinctrl core: initialized pinctrl subsystem [ 0.221200] [ 0.221765] ************************************************************* [ 0.223014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.226011] ** ** [ 0.228009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.230018] ** ** [ 0.233018] ** This means that this kernel is built to expose internal ** [ 0.236016] ** IOMMU data structures, which may compromise security on ** [ 0.238012] ** your system. ** [ 0.240013] ** ** [ 0.243012] ** If you see this message and you are not debugging the ** [ 0.245009] ** kernel, report this immediately to your vendor! ** [ 0.246009] ** ** [ 0.248011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.250009] ************************************************************* [ 0.253774] NET: Registered protocol family 16 [ 0.254519] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.257087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.260136] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.266185] cpuidle: using governor menu [ 0.268012] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.270552] PCI: Using configuration type 1 for base access [ 0.271186] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.282061] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.285022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.291269] cryptd: max_cpu_qlen set to 1000 [ 0.296268] ACPI: Added _OSI(Module Device) [ 0.299014] ACPI: Added _OSI(Processor Device) [ 0.301011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.303011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.307989] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.314781] ACPI: Interpreter enabled [ 0.317066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.320012] ACPI: Using IOAPIC for interrupt routing [ 0.323121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.327555] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.350363] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.355086] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.359025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.361086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.365514] acpiphp: Slot [2] registered [ 0.367158] acpiphp: Slot [5] registered [ 0.368103] acpiphp: Slot [6] registered [ 0.369116] acpiphp: Slot [7] registered [ 0.371125] acpiphp: Slot [8] registered [ 0.372170] acpiphp: Slot [9] registered [ 0.373092] acpiphp: Slot [10] registered [ 0.374000] acpiphp: Slot [3] registered [ 0.375086] acpiphp: Slot [4] registered [ 0.377070] acpiphp: Slot [11] registered [ 0.378070] acpiphp: Slot [12] registered [ 0.379093] acpiphp: Slot [13] registered [ 0.380069] acpiphp: Slot [14] registered [ 0.381073] acpiphp: Slot [15] registered [ 0.383085] acpiphp: Slot [16] registered [ 0.384080] acpiphp: Slot [17] registered [ 0.385072] acpiphp: Slot [18] registered [ 0.386071] acpiphp: Slot [19] registered [ 0.386738] acpiphp: Slot [20] registered [ 0.387081] acpiphp: Slot [21] registered [ 0.388067] acpiphp: Slot [22] registered [ 0.390069] acpiphp: Slot [23] registered [ 0.391078] acpiphp: Slot [24] registered [ 0.392000] acpiphp: Slot [25] registered [ 0.392000] acpiphp: Slot [26] registered [ 0.394100] acpiphp: Slot [27] registered [ 0.395091] acpiphp: Slot [28] registered [ 0.396083] acpiphp: Slot [29] registered [ 0.397000] acpiphp: Slot [30] registered [ 0.397000] acpiphp: Slot [31] registered [ 0.398051] PCI host bridge to bus 0000:00 [ 0.399014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.401022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.403023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.406028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.409035] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.412042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.414208] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.417250] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.421554] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.432989] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.438067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.441022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.443023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.446017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.448709] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.451664] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.454044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.457692] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.463016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.476015] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.480013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.492306] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.507018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.513017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.531018] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.539721] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.544010] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.550015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.560014] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.570235] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.575011] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.578012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.592013] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.604986] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.613016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.618017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.635021] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.646991] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.654025] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.662028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.675014] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.683973] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.688011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.693028] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.708015] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.718736] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.720395] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.724396] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.726430] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728252] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.733054] iommu: Default domain type: Passthrough [ 0.735657] SCSI subsystem initialized [ 0.739186] ACPI: bus type USB registered [ 0.741162] usbcore: registered new interface driver usbfs [ 0.743090] usbcore: registered new interface driver hub [ 0.746108] usbcore: registered new device driver usb [ 0.749204] pps_core: LinuxPPS API ver. 1 registered [ 0.751009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.760113] PTP clock support registered [ 0.763174] EDAC MC: Ver: 3.0.0 [ 0.766357] PCI: Using ACPI for IRQ routing [ 0.767972] NetLabel: Initializing [ 0.771012] NetLabel: domain hash size = 128 [ 0.772009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.775100] NetLabel: unlabeled traffic allowed by default [ 0.781053] vgaarb: loaded [ 0.784096] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.786009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.800653] clocksource: Switched to clocksource kvm-clock [ 1.015949] VFS: Disk quotas dquot_6.6.0 [ 1.017668] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.021365] *** VALIDATE ramfs *** [ 1.026140] *** VALIDATE hugetlbfs *** [ 1.027562] pnp: PnP ACPI init [ 1.029837] pnp: PnP ACPI: found 6 devices [ 1.054241] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.057353] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.059499] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.061577] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.067412] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.069707] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.074481] NET: Registered protocol family 2 [ 1.076979] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.084973] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.094801] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.100582] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.104299] TCP: Hash tables configured (established 65536 bind 65536) [ 1.107630] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.111297] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.114305] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.117471] NET: Registered protocol family 1 [ 1.121866] RPC: Registered named UNIX socket transport module. [ 1.124053] RPC: Registered udp transport module. [ 1.125769] RPC: Registered tcp transport module. [ 1.127355] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.129920] NET: Registered protocol family 44 [ 1.131554] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.133473] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.135199] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.137542] PCI: CLS 0 bytes, default 64 [ 1.139813] Unpacking initramfs... [ 2.722365] debug: unmapping init [mem 0xffff9ae0fcc54000-0xffff9ae0fffbffff] [ 2.726555] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.728487] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.731122] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.243870] Initialise system trusted keyrings [ 3.245288] Key type blacklist registered [ 3.247759] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.256892] zbud: loaded [ 3.260765] *** VALIDATE nfs *** [ 3.261835] *** VALIDATE nfs4 *** [ 3.263375] pstore: using deflate compression [ 3.266644] Platform Keyring initialized [ 3.416981] NET: Registered protocol family 38 [ 3.418721] Key type asymmetric registered [ 3.420017] Asymmetric key parser 'x509' registered [ 3.421649] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.430930] io scheduler mq-deadline registered [ 3.432604] io scheduler kyber registered [ 3.434246] io scheduler bfq registered [ 3.435833] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.440440] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.445378] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.448323] ACPI: Power Button [PWRF] [ 3.555633] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.660945] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.857678] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.951641] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.143619] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.177280] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.209256] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.218482] Non-volatile memory driver v1.3 [ 4.220240] Linux agpgart interface v0.103 [ 4.257375] virtio_blk virtio1: [vda] 133800 512-byte logical blocks (68.5 MB/65.3 MiB) [ 4.261690] vda: detected capacity change from 0 to 68505600 [ 4.287369] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.290257] vdb: detected capacity change from 0 to 1073741824 [ 4.310303] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.314305] vdc: detected capacity change from 0 to 2621440000 [ 4.335526] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.340701] vdd: detected capacity change from 0 to 2621440000 [ 4.360082] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.363943] vde: detected capacity change from 0 to 4294967296 [ 4.383426] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.386439] vdf: detected capacity change from 0 to 4294967296 [ 4.394646] libphy: Fixed MDIO Bus: probed [ 4.409895] usbcore: registered new interface driver usbserial_generic [ 4.412841] usbserial: USB Serial support registered for generic [ 4.416030] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.419985] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.421816] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.424254] mousedev: PS/2 mouse device common for all mice [ 4.427121] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.430565] rtc_cmos 00:05: RTC can wake from S4 [ 4.433318] rtc_cmos 00:05: registered as rtc0 [ 4.440336] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.442566] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.446156] intel_pstate: CPU model not supported [ 4.457232] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.459905] hid: raw HID events driver (C) Jiri Kosina [ 4.467578] usbcore: registered new interface driver usbhid [ 4.469677] usbhid: USB HID core driver [ 4.471333] drop_monitor: Initializing network drop monitor service [ 4.473781] Initializing XFRM netlink socket [ 4.475782] NET: Registered protocol family 10 [ 4.479486] Segment Routing with IPv6 [ 4.482513] NET: Registered protocol family 17 [ 4.484488] mpls_gso: MPLS GSO support [ 4.489415] RAS: Correctable Errors collector initialized. [ 4.491360] AVX version of gcm_enc/dec engaged. [ 4.493045] AES CTR mode by8 optimization enabled [ 4.604659] sched_clock: Marking stable (4604639679, 0)->(5661047451, -1056407772) [ 4.608847] registered taskstats version 1 [ 4.610715] Loading compiled-in X.509 certificates [ 4.614852] zswap: loaded using pool lzo/zbud [ 4.647508] Key type big_key registered [ 4.668952] Key type encrypted registered [ 4.670679] ima: No TPM chip found, activating TPM-bypass! [ 4.672730] ima: Allocated hash algorithm: sha1 [ 4.674429] ima: No architecture policies found [ 4.676265] evm: Initialising EVM extended attributes: [ 4.678242] evm: security.selinux [ 4.679484] evm: security.ima [ 4.680677] evm: security.capability [ 4.681993] evm: HMAC attrs: 0x1 [ 4.685635] rtc_cmos 00:05: setting system clock to 2025-10-07 06:51:20 UTC (1759819880) [ 4.692851] debug: unmapping init [mem 0xffffffffa2803000-0xffffffffa29fffff] [ 4.695931] debug: unmapping init [mem 0xffffffffa1582000-0xffffffffa1858fff] [ 4.703284] Write protecting the kernel read-only data: 28672k [ 4.706834] debug: unmapping init [mem 0xffffffff9fc03000-0xffffffff9fdfffff] [ 4.709672] debug: unmapping init [mem 0xffffffffa0514000-0xffffffffa05fffff] [ 4.758379] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.766909] systemd[1]: Detected virtualization kvm. [ 4.768863] systemd[1]: Detected architecture x86-64. [ 4.771665] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.804942] systemd[1]: No hostname configured. [ 4.807324] systemd[1]: Set hostname to . [ 4.809319] random: systemd: uninitialized urandom read (16 bytes read) [ 4.811740] systemd[1]: Initializing machine ID from random generator. [ 4.857489] random: ln: uninitialized urandom read (6 bytes read) [ 5.060960] random: systemd: uninitialized urandom read (16 bytes read) [ 5.063615] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.069773] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.072995] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.887495] device-mapper: uevent: version 1.0.3 [ 5.889794] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 6.522361] hrtimer: interrupt took 115051225 ns [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted Kernel Configuration File System. Starting dracut initqueue hook...[ 8.594430] random: fast init done [ 8.607409] scsi host0: ata_piix [ 8.619230] scsi host1: ata_piix [ 8.620735] virtio_net virtio0 ens2: renamed from eth0 [ 8.769913] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.769922] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.933191] dracut-initqueue[486]: RTNETLINK answers: File exists [ 16.195949] random: crng init done [ 16.198856] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 17.684546] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.512183] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.047369] SELinux: Disabled at runtime. [ 20.157869] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.172976] systemd[1]: Detected virtualization kvm. [ 20.174838] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.297940] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.305027] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.322574] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.327212] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.330144] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.363959] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.398814] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ 21.563059] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 22.498161] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.211065] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 23.276303] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.970629] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.055484] EDAC sbridge: Ver: 1.1.2 [ 27.236516] Key type dns_resolver registered [ 27.996726] NFS: Registering the id_resolver key type [ 27.998738] Key type id_resolver registered [ 28.000275] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg119-server login: [ 63.708534] libcfs: loading out-of-tree module taints kernel. [ 63.716966] Key type ._llcrypt registered [ 63.718526] Key type .llcrypt registered [ 63.761371] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_hostid [ 70.959908] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 71.646136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 71.653295] alg: No test for adler32 (adler32-zlib) [ 72.649410] Lustre: Lustre: Build Version: 2.16.58_105_g234996a [ 72.942218] LNet: Added LNI 192.168.201.119@tcp [8/256/0/180] [ 74.551229] Key type lgssc registered [ 75.231326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 82.280761] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 83.633486] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.611582] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 90.356145] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.057108] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.640688] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing load_modules_local [ 103.242278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 103.263392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 103.272923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 104.371117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 104.385902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 104.433089] Lustre: lustre-MDT0000: new disk, initializing [ 104.464710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 104.472498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 105.967372] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.190400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 111.222466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 111.249933] Lustre: 6483:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 111.261075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 111.262721] Lustre: Skipped 1 previous similar message [ 111.297946] Lustre: lustre-MDT0001: new disk, initializing [ 111.316682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 111.323121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 111.326268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 112.566606] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 114.894237] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 118.032929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 118.056795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 118.144128] Lustre: lustre-OST0000: new disk, initializing [ 118.145979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 118.162368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 119.989685] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 120.045835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 120.049279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 120.086142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 125.621150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 125.645277] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 125.691912] Lustre: lustre-OST0001: new disk, initializing [ 125.693941] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 125.715994] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 125.875984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 125.879432] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 125.893733] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 127.819653] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 133.964582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 139.658863] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 146.901188] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing check_logdir /tmp/testlogs/ [ 148.665533] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing yml_node [ 150.343406] Lustre: DEBUG MARKER: Client: 2.16.58.105 [ 151.231565] Lustre: DEBUG MARKER: MDS: 2.16.58.105 [ 152.080367] Lustre: DEBUG MARKER: OSS: 2.16.58.105 [ 152.630373] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Tue Oct 7 02:53:48 EDT 2025 [ 158.979112] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 159.713842] Lustre: DEBUG MARKER: === replay-single: start setup 02:53:55 (1759820035) === [ 161.061057] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing check_config_client /mnt/lustre [ 168.007537] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 169.298414] Lustre: 13151:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 170.533763] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.789921] Lustre: DEBUG MARKER: === replay-single: finish setup 02:54:08 (1759820048) === [ 173.463684] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 02:54:09 (1759820049) [ 176.308904] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 176.981843] Lustre: Failing over lustre-MDT0000 [ 177.041489] LustreError: 14020:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 177.063882] Lustre: server umount lustre-MDT0000 complete [ 177.120572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 182.239494] LustreError: 6493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 182.243817] LustreError: 6493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 182.898578] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 187.359649] LustreError: 6493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 187.364328] LustreError: 6493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 191.415832] LDISKFS-fs (dm-0): recovery complete [ 191.417419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.433089] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 191.505628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.019629] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.138548] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 196.578137] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 196.589361] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 197.959149] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 198.635943] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 202.528671] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 02:54:38 (1759820078) [ 203.297267] Lustre: Failing over lustre-OST0000 [ 203.319169] LustreError: 15452:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 203.321693] LustreError: 15452:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 203.342354] Lustre: server umount lustre-OST0000 complete [ 203.388047] LustreError: 8744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 203.393753] LustreError: 8744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 206.815910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 206.816383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 206.826376] Lustre: Skipped 4 previous similar messages [ 208.498654] LustreError: 8744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 208.504814] LustreError: 8744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 216.642940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 216.714765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.349339] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 218.742605] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 218.742615] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 218.749198] Lustre: Skipped 3 previous similar messages [ 218.871167] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.322657] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 222.962237] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 227.087116] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 02:55:02 (1759820102) [ 230.163924] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 231.066712] Lustre: Failing over lustre-MDT0000 [ 231.142549] LustreError: 17150:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 231.144730] LustreError: 17150:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 231.182044] Lustre: server umount lustre-MDT0000 complete [ 232.415860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 232.416235] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 232.416520] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 232.416525] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 232.431291] Lustre: Skipped 3 previous similar messages [ 245.950031] LDISKFS-fs (dm-0): recovery complete [ 245.952237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 245.985511] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 246.057978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 247.532885] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.327595] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 248.330905] Lustre: lustre-MDT0000: Denying connection for new client 231b4762-fae0-4d2c-b717-d0f41e739469 (at 192.168.201.19@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 251.361368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 251.365671] Lustre: Skipped 1 previous similar message [ 253.554843] Lustre: lustre-MDT0000: Denying connection for new client 231b4762-fae0-4d2c-b717-d0f41e739469 (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 253.560594] Lustre: Skipped 1 previous similar message [ 258.674517] Lustre: lustre-MDT0000: Denying connection for new client 231b4762-fae0-4d2c-b717-d0f41e739469 (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 263.794597] Lustre: lustre-MDT0000: Denying connection for new client 231b4762-fae0-4d2c-b717-d0f41e739469 (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 268.914826] Lustre: lustre-MDT0000: Denying connection for new client 231b4762-fae0-4d2c-b717-d0f41e739469 (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 279.154331] Lustre: lustre-MDT0000: Denying connection for new client 231b4762-fae0-4d2c-b717-d0f41e739469 (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:38 [ 279.160014] Lustre: Skipped 1 previous similar message [ 299.635622] Lustre: lustre-MDT0000: Denying connection for new client 231b4762-fae0-4d2c-b717-d0f41e739469 (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 299.643768] Lustre: Skipped 3 previous similar messages [ 318.002283] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 318.006322] Lustre: 17804:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 44c89e60-6142-4b45-8e2d-6f815d637936@ [ 318.012129] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 318.020844] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 318.020923] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 318.030985] Lustre: Skipped 2 previous similar messages [ 318.040184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 318.040199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 325.415676] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 02:56:40 (1759820200) [ 329.084975] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 330.031440] Lustre: Failing over lustre-MDT0000 [ 330.096446] LustreError: 18859:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 330.098710] LustreError: 18859:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 330.128945] Lustre: server umount lustre-MDT0000 complete [ 333.280901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.280984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 333.281782] LustreError: 18161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.281791] LustreError: 18161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 333.289974] Lustre: Skipped 3 previous similar messages [ 345.693965] LDISKFS-fs (dm-0): recovery complete [ 345.696588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.757807] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.849227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.609940] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 348.616086] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 348.622122] Lustre: lustre-MDT0000: Denying connection for new client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 348.631505] Lustre: Skipped 3 previous similar messages [ 351.204620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 415.346533] Lustre: lustre-MDT0000: Denying connection for new client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:02 [ 415.354936] Lustre: Skipped 13 previous similar messages [ 418.000149] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 418.002995] Lustre: 19512:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 231b4762-fae0-4d2c-b717-d0f41e739469@ [ 418.008083] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 418.013795] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 418.014047] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 418.015544] Lustre: Skipped 2 previous similar messages [ 418.032132] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 418.032187] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 424.987636] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 02:58:20 (1759820300) [ 428.012615] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 428.815477] Lustre: Failing over lustre-MDT0000 [ 428.888447] LustreError: 20570:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 428.891215] LustreError: 20570:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 428.925483] Lustre: server umount lustre-MDT0000 complete [ 430.715731] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.723302] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 433.119987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.124374] Lustre: Skipped 2 previous similar messages [ 443.728267] LDISKFS-fs (dm-0): recovery complete [ 443.730523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 443.772773] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.856069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.515301] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 446.066411] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 448.993641] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 449.010359] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 449.031056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 449.031089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 450.417560] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 451.084327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 454.996732] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 02:58:50 (1759820330) [ 457.761139] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 458.422655] Lustre: Failing over lustre-MDT0000 [ 458.596082] LustreError: 22390:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 458.598293] LustreError: 22390:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 458.624256] Lustre: server umount lustre-MDT0000 complete [ 459.232847] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.232974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 459.236499] Lustre: Skipped 4 previous similar messages [ 473.193904] LDISKFS-fs (dm-0): recovery complete [ 473.195490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 473.234875] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.747617] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 476.786932] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 478.716700] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 478.741981] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 478.742095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 480.103612] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 480.757389] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 484.578121] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 02:59:20 (1759820360) [ 487.576559] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 488.370914] Lustre: Failing over lustre-MDT0000 [ 488.529698] LustreError: 24212:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 488.532699] LustreError: 24212:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 488.576375] Lustre: server umount lustre-MDT0000 complete [ 488.928893] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.928985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 488.933897] Lustre: Skipped 3 previous similar messages [ 497.266679] LustreError: 18161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.274655] LustreError: 18161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [ 503.763439] LDISKFS-fs (dm-0): recovery complete [ 503.766413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.819945] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.577877] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 507.506200] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 509.410549] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 509.413619] Lustre: Skipped 7 previous similar messages [ 509.446882] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 509.467493] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:193) [ 509.467935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 510.840519] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 511.489954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 515.523230] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 02:59:51 (1759820391) [ 518.945482] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 519.791552] Lustre: Failing over lustre-MDT0000 [ 519.904497] Lustre: server umount lustre-MDT0000 complete [ 524.768407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.768675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 524.772955] Lustre: Skipped 3 previous similar messages [ 535.183492] LDISKFS-fs (dm-0): recovery complete [ 535.186035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.265331] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.136344] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 540.694568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 540.694745] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 542.269554] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 542.961548] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 547.079864] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 03:00:22 (1759820422) [ 550.296088] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 551.143223] Lustre: Failing over lustre-MDT0000 [ 551.391547] LustreError: 27857:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 551.394707] LustreError: 27857:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 551.432674] Lustre: server umount lustre-MDT0000 complete [ 556.000267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 566.805957] LDISKFS-fs (dm-0): recovery complete [ 566.808508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.744499] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 568.949708] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 568.952554] Lustre: Skipped 1 previous similar message [ 572.407051] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 572.410945] Lustre: Skipped 1 previous similar message [ 572.430029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 572.430177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 573.976083] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 574.677496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 578.880697] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 03:00:54 (1759820454) [ 579.331337] Lustre: *** cfs_fail_loc=13b, val=315*** [ 579.333560] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 579.335026] LustreError: 25215:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae16b5cd500 x1845304954575104/t38654705666(0) o35->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:26/0 lens 392/456 e 0 to 0 dl 1759820466 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 583.563304] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 584.311942] Lustre: Failing over lustre-MDT0000 [ 584.478877] Lustre: server umount lustre-MDT0000 complete [ 599.546977] LDISKFS-fs (dm-0): recovery complete [ 599.548827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.595909] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.600652] LustreError: Skipped 1 previous similar message [ 599.688602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 599.692844] Lustre: Skipped 4 previous similar messages [ 601.251148] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 605.153851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 605.157253] Lustre: Skipped 11 previous similar messages [ 605.168309] Lustre: 25215:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae046185f80 x1845304954575104/t38654705666(0) o35->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:51/0 lens 392/456 e 0 to 0 dl 1759820491 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 605.182040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 605.182049] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 606.559184] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 607.290527] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 611.308573] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 03:01:26 (1759820486) [ 614.162068] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 614.918052] Lustre: Failing over lustre-MDT0000 [ 615.128932] Lustre: server umount lustre-MDT0000 complete [ 615.392120] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.395434] Lustre: Skipped 11 previous similar messages [ 615.396648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 615.400760] LustreError: Skipped 1 previous similar message [ 625.631664] LustreError: 23390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.639312] LustreError: 23390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 62 previous similar messages [ 630.094031] LDISKFS-fs (dm-0): recovery complete [ 630.096080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.241092] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 631.860861] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 635.390158] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 635.390168] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 636.802990] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 637.487986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 641.706185] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 03:01:57 (1759820517) [ 644.930541] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 645.311381] Lustre: *** cfs_fail_loc=114, val=0*** [ 646.426430] Lustre: Failing over lustre-MDT0000 [ 646.688409] LustreError: 33480:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 646.691678] LustreError: 33480:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 646.726128] Lustre: server umount lustre-MDT0000 complete [ 661.948755] LDISKFS-fs (dm-0): recovery complete [ 661.951241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 662.115268] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 662.130568] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 662.134419] Lustre: Skipped 2 previous similar messages [ 663.748610] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 667.119322] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 667.122419] Lustre: Skipped 2 previous similar messages [ 667.138090] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 667.138553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 668.539535] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 669.257953] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 673.182143] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 03:02:28 (1759820548) [ 676.283716] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 676.693565] Lustre: *** cfs_fail_loc=128, val=0*** [ 677.797573] Lustre: Failing over lustre-MDT0000 [ 677.999930] Lustre: server umount lustre-MDT0000 complete [ 682.464193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 682.468404] LustreError: Skipped 1 previous similar message [ 693.027810] LDISKFS-fs (dm-0): recovery complete [ 693.029994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.071425] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 693.077158] LustreError: Skipped 2 previous similar messages [ 693.211167] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 694.891658] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 698.371898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 698.371969] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 699.944682] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 700.645501] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 704.621895] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 03:03:00 (1759820580) [ 707.591649] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 708.445546] Lustre: Failing over lustre-MDT0000 [ 708.576915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.579701] Lustre: Skipped 3 previous similar messages [ 708.741373] Lustre: server umount lustre-MDT0000 complete [ 723.723855] LDISKFS-fs (dm-0): recovery complete [ 723.725535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 723.846722] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 725.514528] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 729.158900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 729.158915] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 730.623100] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 731.263681] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 735.169403] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 03:03:30 (1759820610) [ 738.114658] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 738.808788] Lustre: Failing over lustre-MDT0000 [ 738.945473] Lustre: server umount lustre-MDT0000 complete [ 753.393180] LDISKFS-fs (dm-0): recovery complete [ 753.394780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.523506] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 755.051678] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 758.753581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 758.755208] Lustre: Skipped 19 previous similar messages [ 758.789591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 758.789642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 759.933100] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 760.510904] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 764.000682] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 03:03:59 (1759820639) [ 766.910042] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 769.249562] Lustre: Failing over lustre-MDT0000 [ 769.508024] Lustre: server umount lustre-MDT0000 complete [ 774.111470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.114780] Lustre: Skipped 18 previous similar messages [ 783.615983] LDISKFS-fs (dm-0): recovery complete [ 783.617656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 783.743694] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 784.990321] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 789.592599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 789.592611] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 790.823787] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 791.426430] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 800.774781] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 03:04:36 (1759820676) [ 803.528300] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 804.222580] Lustre: Failing over lustre-MDT0000 [ 804.320258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 804.322225] Lustre: Skipped 3 previous similar messages [ 804.332552] LustreError: 42706:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 804.336650] LustreError: 42706:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 804.370064] Lustre: server umount lustre-MDT0000 complete [ 818.789426] LDISKFS-fs (dm-0): recovery complete [ 818.790994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.948657] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 820.477549] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 820.850403] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 820.854816] Lustre: Skipped 4 previous similar messages [ 824.302643] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 824.306069] Lustre: Skipped 4 previous similar messages [ 824.321737] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 824.321764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 825.456217] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 825.996398] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 831.413664] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 03:05:07 (1759820707) [ 834.035629] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 834.705146] Lustre: Failing over lustre-MDT0000 [ 834.889124] Lustre: server umount lustre-MDT0000 complete [ 839.647601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 839.650576] LustreError: Skipped 3 previous similar messages [ 849.263404] LDISKFS-fs (dm-0): recovery complete [ 849.264848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.305056] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 849.308236] LustreError: Skipped 4 previous similar messages [ 850.780709] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 854.523608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 854.524082] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 855.661259] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 856.203469] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 859.688651] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 03:05:35 (1759820735) [ 862.402614] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 863.105723] Lustre: Failing over lustre-MDT0000 [ 863.259618] Lustre: server umount lustre-MDT0000 complete [ 877.452277] LDISKFS-fs (dm-0): recovery complete [ 877.454249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.561294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.564085] Lustre: Skipped 8 previous similar messages [ 877.602795] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 877.605399] Lustre: Skipped 1 previous similar message [ 878.928638] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 882.685618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 882.685682] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 883.930931] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 884.491195] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 888.060825] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 03:06:03 (1759820763) [ 890.680780] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 891.376157] Lustre: Failing over lustre-MDT0000 [ 891.513098] Lustre: server umount lustre-MDT0000 complete [ 892.531698] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 892.536904] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 105 previous similar messages [ 905.646956] LDISKFS-fs (dm-0): recovery complete [ 905.649476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 907.109497] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 910.857746] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 910.857747] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 912.144504] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 912.783590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 916.327427] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 03:06:32 (1759820792) [ 919.030620] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 919.703962] Lustre: Failing over lustre-MDT0000 [ 919.872034] Lustre: server umount lustre-MDT0000 complete [ 934.032061] LDISKFS-fs (dm-0): recovery complete [ 934.033883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 935.507305] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 939.516245] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 939.516249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 940.816707] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 941.399380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 944.815858] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 03:07:00 (1759820820) [ 947.421695] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 948.072220] Lustre: Failing over lustre-MDT0000 [ 948.224668] Lustre: server umount lustre-MDT0000 complete [ 962.442159] LDISKFS-fs (dm-0): recovery complete [ 962.444107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.564120] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 962.566977] Lustre: Skipped 2 previous similar messages [ 963.785969] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 967.673568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 967.673567] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 968.762713] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 969.242792] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 972.476682] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 03:07:28 (1759820848) [ 974.828663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 975.421441] Lustre: Failing over lustre-MDT0000 [ 975.634704] Lustre: server umount lustre-MDT0000 complete [ 989.920043] LDISKFS-fs (dm-0): recovery complete [ 989.921582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 991.453302] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 995.339030] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 995.339059] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 996.574690] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 997.136441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1000.673832] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 03:07:56 (1759820876) [ 1003.343782] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1004.025173] Lustre: Failing over lustre-MDT0000 [ 1004.269490] Lustre: server umount lustre-MDT0000 complete [ 1018.462578] LDISKFS-fs (dm-0): recovery complete [ 1018.464331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1019.946589] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1023.974703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1023.978434] Lustre: Skipped 35 previous similar messages [ 1024.000982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1024.000995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1025.243772] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1025.868953] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1029.406223] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 03:08:25 (1759820905) [ 1032.070429] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1032.727324] Lustre: Failing over lustre-MDT0000 [ 1032.915610] Lustre: server umount lustre-MDT0000 complete [ 1034.208095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.212099] Lustre: Skipped 36 previous similar messages [ 1046.924817] LDISKFS-fs (dm-0): recovery complete [ 1046.926147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.317627] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1052.150612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 1052.151174] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 1053.293526] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1053.864458] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1057.202598] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 03:08:52 (1759820932) [ 1059.731476] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1060.396836] Lustre: Failing over lustre-MDT0000 [ 1060.517659] LustreError: 59090:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1060.520235] LustreError: 59090:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages [ 1060.547129] Lustre: server umount lustre-MDT0000 complete [ 1074.694228] LDISKFS-fs (dm-0): recovery complete [ 1074.695737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.175052] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1076.850338] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1076.853294] Lustre: Skipped 8 previous similar messages [ 1080.311684] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 1080.311726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:897) [ 1081.524941] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1082.114929] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1085.638423] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 03:09:21 (1759820961) [ 1088.212423] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1088.862115] Lustre: Failing over lustre-MDT0000 [ 1089.119847] Lustre: server umount lustre-MDT0000 complete [ 1103.031939] LDISKFS-fs (dm-0): recovery complete [ 1103.034535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.166302] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1103.168328] Lustre: Skipped 4 previous similar messages [ 1104.375151] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1108.461371] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 1108.465768] Lustre: Skipped 9 previous similar messages [ 1108.483536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1108.483543] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1109.608128] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1110.136914] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1113.593799] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 03:09:49 (1759820989) [ 1116.066553] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1116.719063] Lustre: Failing over lustre-MDT0000 [ 1116.953411] Lustre: server umount lustre-MDT0000 complete [ 1118.688192] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1118.690410] LustreError: Skipped 9 previous similar messages [ 1131.073250] LDISKFS-fs (dm-0): recovery complete [ 1131.074725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.110156] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1131.112959] LustreError: Skipped 9 previous similar messages [ 1132.535099] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1136.635155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:961) [ 1136.635177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:931 to 0x2c0000401:961) [ 1137.899495] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1138.440534] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1141.887731] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 03:10:17 (1759821017) [ 1144.441308] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1145.121958] Lustre: Failing over lustre-MDT0000 [ 1145.275037] Lustre: server umount lustre-MDT0000 complete [ 1159.319844] LDISKFS-fs (dm-0): recovery complete [ 1159.321650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.765381] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1164.790856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 1164.790856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:993) [ 1165.989131] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1166.526575] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1169.954907] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 03:10:45 (1759821045) [ 1172.516987] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1173.225874] Lustre: Failing over lustre-MDT0000 [ 1173.397894] Lustre: server umount lustre-MDT0000 complete [ 1187.449441] LDISKFS-fs (dm-0): recovery complete [ 1187.451297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.885603] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1192.961075] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1192.961076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1194.177764] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1194.716730] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1198.207848] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 03:11:13 (1759821073) [ 1200.731694] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1201.420478] Lustre: Failing over lustre-MDT0000 [ 1201.662949] Lustre: server umount lustre-MDT0000 complete [ 1215.794903] LDISKFS-fs (dm-0): recovery complete [ 1215.796314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.366569] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1221.115527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1221.115531] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1222.325427] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1222.929881] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1226.310254] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 03:11:42 (1759821102) [ 1228.857186] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1229.515862] Lustre: Failing over lustre-MDT0000 [ 1229.649119] Lustre: server umount lustre-MDT0000 complete [ 1243.804107] LDISKFS-fs (dm-0): recovery complete [ 1243.805634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1245.316911] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1249.274430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 1249.274451] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 1250.431186] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1250.988889] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1254.433277] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 03:12:10 (1759821130) [ 1255.681788] Lustre: 71494:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a3bb42a3-5116-47de-8ca5-8789ceee46eb at adminstrative request [ 1257.924791] Lustre: Failing over lustre-MDT0000 [ 1258.163234] Lustre: server umount lustre-MDT0000 complete [ 1271.185524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1272.795279] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1276.404953] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 1276.405025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1277.548912] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1278.137921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1280.496746] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1283.087478] Lustre: DEBUG MARKER: before 3128, after 3128 [ 1285.186400] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 03:12:40 (1759821160) [ 1285.542263] Lustre: 73337:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a3bb42a3-5116-47de-8ca5-8789ceee46eb at adminstrative request [ 1288.986435] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 03:12:44 (1759821164) [ 1291.711212] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1292.460770] Lustre: Failing over lustre-MDT0000 [ 1292.713349] Lustre: server umount lustre-MDT0000 complete [ 1306.772252] LDISKFS-fs (dm-0): recovery complete [ 1306.773914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.079416] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1312.265663] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 1312.265664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 1313.349612] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1313.845223] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1317.006194] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 03:13:12 (1759821192) [ 1319.392887] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1320.022053] Lustre: Failing over lustre-MDT0000 [ 1320.189686] Lustre: server umount lustre-MDT0000 complete [ 1334.180194] LDISKFS-fs (dm-0): recovery complete [ 1334.181832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.560538] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1339.385652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1339.385665] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1340.492235] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1341.049132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1344.397344] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 03:13:40 (1759821220) [ 1346.831508] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1347.451437] Lustre: Failing over lustre-MDT0000 [ 1347.618939] Lustre: server umount lustre-MDT0000 complete [ 1361.854867] LDISKFS-fs (dm-0): recovery complete [ 1361.856384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.999036] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1362.001906] Lustre: Skipped 8 previous similar messages [ 1363.376844] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1367.036372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1367.036422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1368.204470] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1368.775075] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1372.321592] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 03:14:08 (1759821248) [ 1374.942572] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1375.671832] Lustre: Failing over lustre-MDT0000 [ 1375.809202] Lustre: server umount lustre-MDT0000 complete [ 1390.149664] LDISKFS-fs (dm-0): recovery complete [ 1390.151053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.264753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.267022] Lustre: Skipped 17 previous similar messages [ 1391.693621] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1395.707857] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1395.707870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1396.897149] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1397.442431] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1400.910046] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 03:14:36 (1759821276) [ 1403.557952] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1404.225342] Lustre: Failing over lustre-MDT0000 [ 1404.467890] Lustre: server umount lustre-MDT0000 complete [ 1404.532370] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1404.537573] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 263 previous similar messages [ 1418.676908] LDISKFS-fs (dm-0): recovery complete [ 1418.678747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1420.196454] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1423.859499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 1423.859512] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 1425.058876] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1425.686692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1429.082479] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 03:15:04 (1759821304) [ 1431.583391] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1432.288201] Lustre: Failing over lustre-MDT0000 [ 1432.383585] Lustre: server umount lustre-MDT0000 complete [ 1446.521282] LDISKFS-fs (dm-0): recovery complete [ 1446.523518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.006806] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1452.036921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1452.036922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1453.222757] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1453.799785] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1457.398936] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 03:15:33 (1759821333) [ 1460.103696] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1460.835436] Lustre: Failing over lustre-MDT0000 [ 1460.851596] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 1461.028192] Lustre: server umount lustre-MDT0000 complete [ 1475.153032] LDISKFS-fs (dm-0): recovery complete [ 1475.154436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1476.528504] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1480.713752] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1480.713752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1481.969251] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1482.553318] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1486.203464] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 03:16:01 (1759821361) [ 1488.828550] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1489.495956] Lustre: Failing over lustre-MDT0000 [ 1489.725057] Lustre: server umount lustre-MDT0000 complete [ 1503.951603] LDISKFS-fs (dm-0): recovery complete [ 1503.953653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.437430] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1509.369418] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1509.369423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1510.571417] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1511.098171] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1514.611397] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 03:16:30 (1759821390) [ 1517.134454] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1517.799655] Lustre: Failing over lustre-MDT0000 [ 1518.017620] Lustre: server umount lustre-MDT0000 complete [ 1532.253390] LDISKFS-fs (dm-0): recovery complete [ 1532.255381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.748696] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1537.505651] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1537.508181] Lustre: Skipped 71 previous similar messages [ 1537.527662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1537.527675] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1538.740314] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1539.337599] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1542.891383] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 03:16:58 (1759821418) [ 1545.515528] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1546.186047] Lustre: Failing over lustre-MDT0000 [ 1546.328690] Lustre: server umount lustre-MDT0000 complete [ 1547.743887] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.748526] Lustre: Skipped 71 previous similar messages [ 1560.477913] LDISKFS-fs (dm-0): recovery complete [ 1560.479419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.954332] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1565.684135] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1565.684134] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1566.855529] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1567.454737] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1570.971402] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 03:17:26 (1759821446) [ 1573.483133] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1574.185535] Lustre: Failing over lustre-MDT0000 [ 1574.356984] LustreError: 92234:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1574.359830] LustreError: 92234:0:(obd_class.h:479:obd_check_dev()) Skipped 143 previous similar messages [ 1574.386945] Lustre: server umount lustre-MDT0000 complete [ 1588.536746] LDISKFS-fs (dm-0): recovery complete [ 1588.539399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.850396] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1588.853836] Lustre: Skipped 17 previous similar messages [ 1589.943145] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1593.852084] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 1593.852090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 1595.025643] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1595.558260] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1599.008572] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 03:17:54 (1759821474) [ 1599.342091] Lustre: 93611:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a3bb42a3-5116-47de-8ca5-8789ceee46eb at adminstrative request [ 1603.812965] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 03:17:59 (1759821479) [ 1605.968153] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1606.591706] Lustre: Failing over lustre-MDT0000 [ 1606.795687] Lustre: server umount lustre-MDT0000 complete [ 1610.994380] LDISKFS-fs (dm-0): recovery complete [ 1610.996467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1611.151886] Lustre: lustre-MDT0000: Aborting client recovery [ 1611.154110] LustreError: 94891:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1611.158035] Lustre: 94924:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1611.162102] Lustre: 94924:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a3bb42a3-5116-47de-8ca5-8789ceee46eb@ [ 1611.167622] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1611.171437] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1611.178437] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1611.203091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 1611.203092] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 1612.513854] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1616.353051] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1623.985147] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 03:18:19 (1759821499) [ 1626.427985] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1627.053887] Lustre: Failing over lustre-MDT0000 [ 1627.157378] Lustre: server umount lustre-MDT0000 complete [ 1631.419185] LDISKFS-fs (dm-0): recovery complete [ 1631.421495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.533204] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1631.551697] Lustre: lustre-MDT0000: Aborting client recovery [ 1631.554126] LustreError: 96736:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1631.558036] Lustre: 96769:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1631.560809] Lustre: 96769:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 1631.563167] Lustre: 96769:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1631.566573] Lustre: 96769:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1631.569180] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1631.572585] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 1631.578904] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 1631.601630] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1537) [ 1631.601631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1537) [ 1632.939288] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1636.832539] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1642.384550] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1642.386773] Lustre: Skipped 1 previous similar message [ 1644.650125] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 03:18:40 (1759821520) [ 1647.213616] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1647.829500] Lustre: Failing over lustre-MDT0000 [ 1647.919897] Lustre: server umount lustre-MDT0000 complete [ 1652.173690] LDISKFS-fs (dm-0): recovery complete [ 1652.174970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1652.192327] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1652.197905] LustreError: Skipped 16 previous similar messages [ 1652.211622] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.215535] LustreError: Skipped 18 previous similar messages [ 1652.311957] Lustre: lustre-MDT0000: Aborting client recovery [ 1652.313861] LustreError: 98585:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1652.317045] Lustre: 98618:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1652.320267] Lustre: 98618:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 1652.323100] Lustre: 98618:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1652.326593] Lustre: 98618:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1652.329111] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1652.332450] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 1652.340110] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 1652.364262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 1652.364262] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 1653.717805] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1657.313522] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1665.271650] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 03:19:00 (1759821540) [ 1665.642513] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1665.645304] LustreError: 18161:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae049514700 x1845304955484544/t201863462916(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:357/0 lens 512/456 e 0 to 0 dl 1759821552 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1668.259796] Lustre: Failing over lustre-MDT0000 [ 1668.344285] Lustre: server umount lustre-MDT0000 complete [ 1671.387684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1671.517726] Lustre: lustre-MDT0000: Aborting client recovery [ 1671.519170] LustreError: 99890:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1671.521536] Lustre: 99923:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1671.523837] Lustre: 99923:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 1671.541787] Lustre: 99923:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1671.545462] Lustre: 99923:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1671.547544] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1671.550736] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 1671.556814] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 1671.581361] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1601) [ 1671.581407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1601) [ 1672.873711] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1676.768889] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1685.295084] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1685.829528] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 03:19:21 (1759821561) [ 1688.455231] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1689.385173] Lustre: Failing over lustre-MDT0000 [ 1689.536867] Lustre: server umount lustre-MDT0000 complete [ 1693.834527] LDISKFS-fs (dm-0): recovery complete [ 1693.836322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1694.011920] Lustre: lustre-MDT0000: Aborting client recovery [ 1694.013728] LustreError: 101835:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1694.016069] Lustre: 101868:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1694.020280] Lustre: 101868:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 1694.023107] Lustre: 101868:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1694.027637] Lustre: 101868:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1694.030896] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1694.034141] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 1694.040807] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 1694.065560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1633) [ 1694.066119] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1571 to 0x2c0000401:1633) [ 1695.391350] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1699.297637] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1707.130065] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 03:19:42 (1759821582) [ 1715.927120] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1716.558062] Lustre: Failing over lustre-MDT0000 [ 1716.851500] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 1716.879766] Lustre: server umount lustre-MDT0000 complete [ 1731.143271] LDISKFS-fs (dm-0): recovery complete [ 1731.144707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.602620] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1736.695382] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 1736.698185] Lustre: Skipped 17 previous similar messages [ 1736.712514] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 1736.712537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 1737.887839] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1738.494628] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1746.575459] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 03:20:22 (1759821622) [ 1753.097088] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1755.954050] Lustre: Failing over lustre-MDT0000 [ 1756.181116] Lustre: server umount lustre-MDT0000 complete [ 1770.353812] LDISKFS-fs (dm-0): recovery complete [ 1770.355454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.868249] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1776.212485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 1776.212516] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 1777.308503] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1777.845028] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1785.994508] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 03:21:01 (1759821661) [ 1786.737065] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1787.094797] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1787.098361] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1789.269108] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 03:21:04 (1759821664) [ 1795.745137] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1798.794882] Lustre: Failing over lustre-OST0000 [ 1799.165227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1800.918331] Lustre: server umount lustre-OST0000 complete [ 1815.335646] LDISKFS-fs (dm-2): recovery complete [ 1815.337108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1817.249721] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1862.573645] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 03:22:18 (1759821738) [ 1865.098407] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1865.970232] Lustre: Failing over lustre-MDT0000 [ 1865.976590] LustreError: 3627:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ae049d25c00 x1845304963453440/t0(0) o6->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 1866.110986] Lustre: server umount lustre-MDT0000 complete [ 1880.225788] LDISKFS-fs (dm-0): recovery complete [ 1880.227876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.352329] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1880.355483] Lustre: Skipped 26 previous similar messages [ 1881.724357] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1885.689519] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 1885.689572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 1886.821346] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1887.350617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1900.843693] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 03:22:56 (1759821776) [ 1902.047098] Lustre: 110939:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759821761/real 1759821761] req@ffff9ae15f951500 x1845304963558656/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1759821777 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 1902.055050] LustreError: 110939:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 1902.056411] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1902.552590] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1903.071526] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 1907.679129] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1907.681428] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnecting [ 1908.283773] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1913.311157] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1913.313178] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnecting [ 1913.936650] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1918.943178] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1918.945818] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnecting [ 1919.576964] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1925.087157] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1925.698784] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1930.719139] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1930.721797] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnecting [ 1930.725334] Lustre: Skipped 1 previous similar message [ 1936.966838] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1936.970084] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 1941.983133] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1941.987243] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 1948.127260] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnecting [ 1948.130793] Lustre: Skipped 2 previous similar messages [ 1954.386756] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 1954.389020] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 1959.903102] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 1959.905283] LustreError: 10264:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 1962.747155] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 03:23:58 (1759821838) [ 1963.426227] LustreError: 10264:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1972.850179] Lustre: lustre-MDT0000: Export ffff9ae0472c3000 already connecting from 192.168.201.19@tcp [ 1977.970472] Lustre: lustre-MDT0000: Export ffff9ae0472c3000 already connecting from 192.168.201.19@tcp [ 1983.090370] Lustre: lustre-MDT0000: Export ffff9ae0472c3000 already connecting from 192.168.201.19@tcp [ 1988.210160] Lustre: lustre-MDT0000: Export ffff9ae0472c3000 already connecting from 192.168.201.19@tcp [ 1988.213871] Lustre: Skipped 1 previous similar message [ 1993.330321] Lustre: lustre-MDT0000: Export ffff9ae0472c3000 already connecting from 192.168.201.19@tcp [ 2003.471109] LustreError: 10264:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2003.473467] Lustre: 10264:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9ae1737e0000 x1845304958112896/t0(0) o38->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1759821859 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2003.570406] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnecting [ 2003.573092] Lustre: Skipped 3 previous similar messages [ 2003.574598] LustreError: 6488:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2028.146450] Lustre: lustre-MDT0000: Export ffff9ae0472c3000 already connecting from 192.168.201.19@tcp [ 2028.148737] Lustre: Skipped 1 previous similar message [ 2043.615158] LustreError: 6488:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2043.618563] Lustre: 6488:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9ae04a20fb80 x1845304958118144/t0(0) o38->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1759821899 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2047.155422] LustreError: 6487:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2072.690259] Lustre: lustre-MDT0000: Export ffff9ae0472c3000 already connecting from 192.168.201.19@tcp [ 2072.692515] Lustre: Skipped 3 previous similar messages [ 2087.199122] LustreError: 6487:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2087.203625] Lustre: 6487:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff9ae173609f80 x1845304958121728/t0(0) o38->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1759821942 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2088.050062] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnecting [ 2088.053159] Lustre: Skipped 1 previous similar message [ 2088.054293] LustreError: 6489:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2113.650336] Lustre: lustre-MDT0000: Export ffff9ae0472c3000 already connecting from 192.168.201.19@tcp [ 2113.652580] Lustre: Skipped 2 previous similar messages [ 2128.095169] LustreError: 6489:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2128.097339] Lustre: 6489:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9ae1733f5880 x1845304958124672/t0(0) o38->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1759821983 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2129.010225] LustreError: 6517:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2169.055149] LustreError: 6517:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2169.057371] Lustre: 6517:0:(service.c:2560:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9ae145e65880 x1845304958127744/t0(0) o38->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:0/0 lens 520/416 e 0 to 0 dl 1759822024 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2169.970403] LustreError: 113317:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2173.199054] LustreError: 113317:0:(ldlm_lib.c:1418:target_handle_connect()) cfs_fail_timeout interrupted [ 2174.621794] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 03:27:30 (1759822050) [ 2177.099199] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2178.312202] Lustre: Failing over lustre-MDT0000 [ 2178.383853] LustreError: 115021:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2178.387110] LustreError: 115021:0:(obd_class.h:479:obd_check_dev()) Skipped 75 previous similar messages [ 2178.420354] Lustre: server umount lustre-MDT0000 complete [ 2178.527718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.528553] LustreError: 113317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.532756] Lustre: Skipped 42 previous similar messages [ 2178.536904] LustreError: 113317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 171 previous similar messages [ 2182.793983] LDISKFS-fs (dm-0): recovery complete [ 2182.795371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2182.893548] Lustre: *** cfs_fail_loc=712, val=0*** [ 2182.895449] LustreError: 8744:0:(service.c:1389:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9ae16ac8b800 x1845304963702272/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2182.910669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.912607] Lustre: Skipped 16 previous similar messages [ 2182.966698] Lustre: lustre-MDT0000: Aborting client recovery [ 2182.968160] LustreError: 115665:0:(ldlm_lib.c:2982:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2182.970441] Lustre: 115698:0:(ldlm_lib.c:2385:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2182.972695] Lustre: 115698:0:(ldlm_lib.c:2385:target_recovery_overseer()) Skipped 2 previous similar messages [ 2182.974996] Lustre: 115698:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2182.977680] Lustre: 115698:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2182.981694] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2182.985737] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2182.991281] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2183.013551] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2183.013554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2184.368207] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2188.256760] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2188.264459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2188.266134] Lustre: Skipped 48 previous similar messages [ 2194.416107] Lustre: Failing over lustre-MDT0000 [ 2194.580685] Lustre: server umount lustre-MDT0000 complete [ 2207.406201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.840561] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2210.930183] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2210.932821] Lustre: Skipped 4 previous similar messages [ 2212.857952] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2212.857977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2213.949080] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2214.460897] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2217.906599] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 03:28:13 (1759822093) [ 2217.942900] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnecting [ 2217.946263] Lustre: Skipped 2 previous similar messages [ 2220.999674] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 03:28:16 (1759822096) [ 2221.312122] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2221.313572] LustreError: 100655:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae16ac72680 x1845304958183168/t0(0) o700->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:158/0 lens 264/248 e 0 to 0 dl 1759822108 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2238.697288] Lustre: Failing over lustre-MDT0000 [ 2238.900409] Lustre: server umount lustre-MDT0000 complete [ 2251.937302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.471368] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2257.395269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2257.395275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2258.634130] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2259.202188] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2263.133885] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 03:28:58 (1759822138) [ 2263.902885] Lustre: Failing over lustre-OST0000 [ 2263.953824] Lustre: server umount lustre-OST0000 complete [ 2267.616451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2267.618458] LustreError: Skipped 7 previous similar messages [ 2276.789520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2278.829881] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2281.906121] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2282.447507] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2347.674256] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 03:30:23 (1759822223) [ 2350.239211] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2350.978576] Lustre: Failing over lustre-MDT0000 [ 2351.137187] Lustre: server umount lustre-MDT0000 complete [ 2365.317320] LDISKFS-fs (dm-0): recovery complete [ 2365.318699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.350725] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.353958] LustreError: Skipped 8 previous similar messages [ 2366.726297] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2370.601580] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2370.605265] Lustre: Skipped 6 previous similar messages [ 2370.622113] Lustre: *** cfs_fail_loc=216, val=0*** [ 2370.622399] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2370.623490] LustreError: 121462:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2371.679470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2433.056002] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 03:31:48 (1759822308) [ 2440.947936] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 03:31:56 (1759822316) [ 2441.915494] Lustre: Failing over lustre-MDT0000 [ 2442.058520] Lustre: server umount lustre-MDT0000 complete [ 2455.004416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.517284] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2460.643466] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2460.644862] LustreError: 123025:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae172f36680 x1845304958300928/t0(0) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:438/0 lens 328/344 e 0 to 0 dl 1759822388 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2517.618206] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnected, waiting for 2 clients in recovery for 1:07 [ 2517.625118] Lustre: 123025:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2517.647109] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 2517.647114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 2519.021356] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2519.530954] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2523.191484] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 03:33:18 (1759822398) [ 2524.544623] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2527.305021] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2527.959840] Lustre: Failing over lustre-MDT0000 [ 2528.139493] Lustre: server umount lustre-MDT0000 complete [ 2542.173588] LDISKFS-fs (dm-0): recovery complete [ 2542.175742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.300383] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2542.303259] Lustre: Skipped 8 previous similar messages [ 2543.606132] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2547.702671] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3105) [ 2547.702735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3137) [ 2548.872853] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2549.400581] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2552.823763] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 03:33:48 (1759822428) [ 2553.187192] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2556.996051] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2557.634944] Lustre: Failing over lustre-MDT0000 [ 2557.717307] Lustre: server umount lustre-MDT0000 complete [ 2571.945602] LDISKFS-fs (dm-0): recovery complete [ 2571.947209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2573.309500] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2577.395618] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 2577.395630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3107 to 0x280000401:3169) [ 2578.542197] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2579.095553] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2582.635267] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 03:34:18 (1759822458) [ 2582.984497] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2586.736894] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2587.376169] Lustre: Failing over lustre-MDT0000 [ 2587.458984] Lustre: server umount lustre-MDT0000 complete [ 2601.789946] LDISKFS-fs (dm-0): recovery complete [ 2601.791447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.217950] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2607.096154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 2607.096222] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3169) [ 2611.656784] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 03:34:47 (1759822487) [ 2613.040342] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2613.041646] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2613.042895] LustreError: 6491:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae1431eb850 x1845304958351616/t257698037777(0) o35->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:549/0 lens 392/456 e 0 to 0 dl 1759822499 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2613.947962] Lustre: Failing over lustre-MDT0000 [ 2614.093107] Lustre: server umount lustre-MDT0000 complete [ 2626.961527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.411781] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2632.178496] Lustre: 6491:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae17395e680 x1845304958351616/t257698037777(0) o35->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:568/0 lens 392/456 e 0 to 0 dl 1759822518 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2632.184791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3233) [ 2632.184875] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 2633.339978] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2633.884586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2637.361447] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 03:35:13 (1759822513) [ 2637.725887] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2637.727496] LustreError: 18161:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae04986c700 x1845304958364800/t261993005072(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:619/0 lens 504/448 e 0 to 0 dl 1759822569 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2641.570877] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2642.234403] Lustre: Failing over lustre-MDT0000 [ 2642.323486] Lustre: server umount lustre-MDT0000 complete [ 2656.655848] LDISKFS-fs (dm-0): recovery complete [ 2656.658061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2658.212658] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2661.874234] Lustre: 6487:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae04a1f3b80 x1845304958364800/t261993005072(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:643/0 lens 504/2880 e 0 to 0 dl 1759822593 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2661.880107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3233) [ 2661.880147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3235 to 0x280000401:3265) [ 2663.092077] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2663.644106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2667.183045] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 03:35:42 (1759822542) [ 2667.549368] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2667.551296] LustreError: 113317:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae04a6a6300 x1845304958378496/t266287972368(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:648/0 lens 504/448 e 0 to 0 dl 1759822598 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2668.843577] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2671.415779] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2672.087465] Lustre: Failing over lustre-MDT0000 [ 2672.097831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.100054] Lustre: Skipped 3 previous similar messages [ 2672.183238] Lustre: server umount lustre-MDT0000 complete [ 2686.524748] LDISKFS-fs (dm-0): recovery complete [ 2686.526895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.993369] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2692.083758] Lustre: 25215:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae1493e9c00 x1845304958378752/t266287972369(0) o35->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:628/0 lens 392/456 e 0 to 0 dl 1759822578 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2692.083758] Lustre: 10264:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae1493e8700 x1845304958378496/t266287972368(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:672/0 lens 504/2880 e 0 to 0 dl 1759822622 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2692.089396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3267 to 0x280000401:3297) [ 2692.089405] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3265) [ 2696.581446] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 03:36:12 (1759822572) [ 2696.935071] LustreError: 10264:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae16ac8bb80 x1845304958390784/t270582939664(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:677/0 lens 504/448 e 0 to 0 dl 1759822627 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2696.946020] LustreError: 10264:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 2698.222670] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2698.224497] Lustre: Skipped 1 previous similar message [ 2700.931768] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2701.551049] Lustre: Failing over lustre-MDT0000 [ 2701.648537] Lustre: server umount lustre-MDT0000 complete [ 2715.895840] LDISKFS-fs (dm-0): recovery complete [ 2715.897226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2717.459893] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2721.264360] Lustre: 6487:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae181a8b100 x1845304958390784/t270582939664(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:702/0 lens 504/2880 e 0 to 0 dl 1759822652 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2721.270903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3299 to 0x280000401:3329) [ 2721.270949] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3297) [ 2725.471994] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 03:36:41 (1759822601) [ 2725.844044] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2727.146240] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2727.148042] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2727.149329] Lustre: Skipped 2 previous similar messages [ 2730.641644] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2731.278417] Lustre: Failing over lustre-MDT0000 [ 2731.362436] Lustre: server umount lustre-MDT0000 complete [ 2745.433235] LDISKFS-fs (dm-0): recovery complete [ 2745.434783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.908302] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2750.961973] Lustre: 25215:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae16ac73b80 x1845304958403968/t274877906960(0) o35->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:687/0 lens 392/456 e 0 to 0 dl 1759822637 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2750.971058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 2750.971165] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3329) [ 2755.612770] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 03:37:11 (1759822631) [ 2755.956676] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 2755.958702] LustreError: 6489:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae15f948000 x1845304958414976/t279172874255(0) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:736/0 lens 664/608 e 0 to 0 dl 1759822686 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2755.965076] LustreError: 6489:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 2816.626338] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnecting [ 2816.630204] Lustre: Skipped 4 previous similar messages [ 2816.637647] Lustre: 6488:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae16b5cc380 x1845304958414976/t279172874255(0) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:42/0 lens 664/3488 e 0 to 0 dl 1759822747 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2819.000398] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 03:38:14 (1759822694) [ 2821.594778] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2822.168794] Lustre: Failing over lustre-MDT0000 [ 2822.230989] LustreError: 138955:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2822.233031] LustreError: 138955:0:(obd_class.h:479:obd_check_dev()) Skipped 107 previous similar messages [ 2822.254313] Lustre: server umount lustre-MDT0000 complete [ 2822.623862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.624516] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.627355] Lustre: Skipped 56 previous similar messages [ 2822.630980] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 170 previous similar messages [ 2836.447567] LDISKFS-fs (dm-0): recovery complete [ 2836.448977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.553201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.555778] Lustre: Skipped 13 previous similar messages [ 2837.106057] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2837.108654] Lustre: Skipped 12 previous similar messages [ 2837.918569] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2841.569176] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2841.571893] Lustre: Skipped 54 previous similar messages [ 2841.593582] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3331 to 0x2c0000401:3361) [ 2841.593607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3393) [ 2842.823421] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2843.418023] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2857.038391] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 03:38:52 (1759822732) [ 2859.761178] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2860.389209] Lustre: Failing over lustre-MDT0000 [ 2860.601573] Lustre: server umount lustre-MDT0000 complete [ 2874.590157] LDISKFS-fs (dm-0): recovery complete [ 2874.591472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.975916] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2879.998663] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 2879.998662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3425) [ 2881.188639] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2881.769390] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2884.123413] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2889.201839] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 03:39:24 (1759822764) [ 2898.866913] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2899.522144] Lustre: Failing over lustre-MDT0000 [ 2899.874645] Lustre: server umount lustre-MDT0000 complete [ 2900.448157] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2900.449926] LustreError: Skipped 10 previous similar messages [ 2914.066811] LDISKFS-fs (dm-0): recovery complete [ 2914.068238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.414366] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2919.477140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 2919.477140] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 2920.648503] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2921.211914] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2935.733491] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 03:40:10 (1759822810) [ 2938.940657] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2939.627635] Lustre: Failing over lustre-MDT0000 [ 2939.877346] Lustre: server umount lustre-MDT0000 complete [ 2953.951741] LDISKFS-fs (dm-0): recovery complete [ 2953.953648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.462546] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2959.361832] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 2959.361838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 2960.542580] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2961.133140] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2964.034437] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 2964.566676] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2966.777910] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 03:40:42 (1759822842) [ 2972.631026] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3033.471512] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3033.473022] Lustre: Skipped 1 previous similar message [ 3033.474214] LustreError: 6489:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae151b1cc50 x1845304961204608/t296352743435(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:259/0 lens 66040/440 e 0 to 0 dl 1759822964 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3093.108453] Lustre: 6488:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae173843100 x1845304961204608/t296352743435(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:318/0 lens 66040/440 e 0 to 0 dl 1759823023 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3095.937562] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3096.430379] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 03:42:52 (1759822972) [ 3100.502923] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3101.340355] Lustre: Failing over lustre-MDT0000 [ 3101.550115] Lustre: server umount lustre-MDT0000 complete [ 3115.672283] LDISKFS-fs (dm-0): recovery complete [ 3115.674498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.716529] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3115.719964] LustreError: Skipped 13 previous similar messages [ 3117.230831] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3121.273395] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3121.276988] Lustre: Skipped 13 previous similar messages [ 3121.293199] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4833) [ 3121.293199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4865) [ 3122.472913] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3123.011777] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3126.715572] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 03:43:22 (1759823002) [ 3132.514547] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3135.749306] Lustre: Failing over lustre-OST0000 [ 3135.781745] Lustre: server umount lustre-OST0000 complete [ 3149.769316] LDISKFS-fs (dm-2): recovery complete [ 3149.771456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3149.829921] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3149.832214] Lustre: Skipped 12 previous similar messages [ 3151.711670] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3163.014596] Lustre: Failing over lustre-OST0000 [ 3163.049902] Lustre: server umount lustre-OST0000 complete [ 3175.952922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3177.939277] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3181.137089] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3181.709724] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3215.595352] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 03:44:51 (1759823091) [ 3216.472572] Lustre: Failing over lustre-MDT0000 [ 3216.682959] Lustre: server umount lustre-MDT0000 complete [ 3229.542501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.064816] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3234.809847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4897) [ 3234.809847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4865) [ 3242.627966] Lustre: Failing over lustre-MDT0000 [ 3242.799838] Lustre: server umount lustre-MDT0000 complete [ 3255.765720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.234126] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3260.917623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4807 to 0x2c0000401:4897) [ 3260.917656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4838 to 0x280000401:4929) [ 3262.151340] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3262.745775] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3266.248907] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 03:45:41 (1759823141) [ 3277.637664] Lustre: Failing over lustre-OST0000 [ 3277.671581] Lustre: server umount lustre-OST0000 complete [ 3290.419263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3292.288764] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3295.352548] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3295.863281] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3299.699264] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 03:46:15 (1759823175) [ 3300.340954] Lustre: Failing over lustre-MDT0000 [ 3300.693674] Lustre: server umount lustre-MDT0000 complete [ 3303.922483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3303.961788] Lustre: *** cfs_fail_loc=605, val=0*** [ 3303.963917] LustreError: 156713:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc06fdf60 failed: rc = -95 [ 3303.968724] LustreError: 156713:0:(obd_config.c:783:class_setup()) setup MGS failed (-95) [ 3303.971306] LustreError: 156713:0:(obd_mount.c:193:lustre_start_simple()) MGS setup error -95 [ 3303.974172] LustreError: 156713:0:(tgt_mount.c:117:server_deregister_mount()) MGS not registered [ 3303.977165] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3303.979524] LustreError: 156713:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-MDT0000 [ 3303.983706] Lustre: server umount lustre-MDT0000 complete [ 3303.985119] LustreError: 156713:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3306.604142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.069437] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3310.738762] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 03:46:26 (1759823186) [ 3312.118519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3312.118732] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3314.679124] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3315.704059] Lustre: Failing over lustre-MDT0000 [ 3316.062453] Lustre: server umount lustre-MDT0000 complete [ 3330.186308] LDISKFS-fs (dm-0): recovery complete [ 3330.188810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3331.662147] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3333.747696] Lustre: *** cfs_fail_loc=707, val=0*** [ 3393.147801] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 3393.159968] Lustre: 158865:0:(ldlm_lib.c:2067:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3393.164393] Lustre: 158865:0:(ldlm_lib.c:2067:extend_recovery_timer()) Skipped 1 previous similar message [ 3393.255365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 3393.255364] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 3394.595565] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3395.181576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3398.978547] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 03:47:54 (1759823274) [ 3422.562640] LustreError: 18161:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9ae17395e300 x1845304962083712/t0(0) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:604/0 lens 664/0 e 0 to 0 dl 1759823309 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3422.568896] LustreError: 18161:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3433.591120] LustreError: 18161:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3433.596787] LustreError: 6490:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9ae17313dc00 x1845304962084992/t0(0) o35->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:615/0 lens 392/0 e 0 to 0 dl 1759823320 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3439.072837] LustreError: 40350:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9ae173294a80 x1845304965287680/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:620/0 lens 544/0 e 0 to 0 dl 1759823325 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3439.078722] LustreError: 40350:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 48 previous similar messages [ 3445.453199] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 03:48:41 (1759823321) [ 3469.090706] LustreError: 8377:0:(tgt_handler.c:2784:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3480.119120] LustreError: 8377:0:(tgt_handler.c:2784:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3483.399545] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 03:49:19 (1759823359) [ 3506.396264] LustreError: 6488:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9ae1493e9180 x1845304962108800/t0(0) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:688/0 lens 576/0 e 0 to 0 dl 1759823393 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3506.402740] LustreError: 6488:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3511.503102] LustreError: 6488:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3511.507732] LustreError: 6488:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9ae14f7e9c00 x1845304962109696/t0(0) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:693/0 lens 664/0 e 0 to 0 dl 1759823398 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3522.360067] LustreError: 6488:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3522.364652] LustreError: 6488:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9ae172ffe300 x1845304962131456/t0(0) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:743/0 lens 664/0 e 0 to 0 dl 1759823448 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3522.370755] LustreError: 6488:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 3534.262143] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 03:50:10 (1759823410) [ 3619.167229] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 03:51:34 (1759823494) [ 3642.112464] LustreError: 6517:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9ae15f9d8000 x1845304962188672/t0(0) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:68/0 lens 576/0 e 0 to 0 dl 1759823528 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3642.119242] LustreError: 6517:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 3642.121911] LustreError: 6517:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3642.124777] LustreError: 6517:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 3642.543094] LustreError: 6517:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3658.361222] LustreError: 113317:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3658.363745] LustreError: 113317:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3658.783106] LustreError: 113317:0:(service.c:2537:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3658.786933] LustreError: 113317:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3674.193393] LustreError: 6517:0:(service.c:2536:ptlrpc_server_handle_request()) @@@ HIT req@ffff9ae1733f4380 x1845304962208000/t0(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:101/0 lens 504/0 e 0 to 0 dl 1759823561 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 3674.203484] LustreError: 6517:0:(service.c:2536:ptlrpc_server_handle_request()) Skipped 82 previous similar messages [ 3686.257660] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 03:52:42 (1759823562) [ 3710.602292] Lustre: DEBUG MARKER: phase 2 [ 3713.642702] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 03:53:09 (1759823589) [ 3783.788135] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 03:54:19 (1759823659) [ 3784.217613] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 3784.708398] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 03:54:20 (1759823660) [ 3786.409175] Lustre: DEBUG MARKER: Started rundbench load pid=133474 ... [ 3789.987680] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3791.504743] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 3792.149759] Lustre: Failing over lustre-MDT0000 [ 3792.471338] LustreError: 165761:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3792.473949] LustreError: 165761:0:(obd_class.h:479:obd_check_dev()) Skipped 83 previous similar messages [ 3792.512867] Lustre: server umount lustre-MDT0000 complete [ 3792.716219] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.19@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3792.720917] LustreError: 6487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 150 previous similar messages [ 3792.863518] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3792.867484] LustreError: Skipped 6 previous similar messages [ 3792.869512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3792.874802] Lustre: Skipped 38 previous similar messages [ 3807.019587] LDISKFS-fs (dm-0): recovery complete [ 3807.022062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3807.068878] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3807.074856] LustreError: Skipped 4 previous similar messages [ 3807.157599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.159475] Lustre: Skipped 11 previous similar messages [ 3807.171817] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3807.173639] Lustre: Skipped 6 previous similar messages [ 3808.505815] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3809.905614] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3809.910687] Lustre: Skipped 11 previous similar messages [ 3812.322365] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3812.326782] Lustre: Skipped 41 previous similar messages [ 3812.336787] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3812.340082] Lustre: Skipped 7 previous similar messages [ 3812.355058] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5057) [ 3812.355119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5089) [ 3813.524847] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3814.080532] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3818.980123] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3820.482615] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 3821.127277] Lustre: Failing over lustre-MDT0001 [ 3821.335884] Lustre: server umount lustre-MDT0001 complete [ 3835.288521] LDISKFS-fs (dm-1): recovery complete [ 3835.289724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3836.602774] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3840.882414] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:819 to 0x280000400:865) [ 3840.883346] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:801 to 0x2c0000400:833) [ 3842.001572] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3842.543202] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3847.565673] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3849.092207] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 3849.714577] Lustre: Failing over lustre-MDT0000 [ 3850.090644] Lustre: server umount lustre-MDT0000 complete [ 3864.428246] LDISKFS-fs (dm-0): recovery complete [ 3864.429871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3865.955702] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3869.691723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5121) [ 3869.691729] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5089) [ 3870.948626] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3871.511200] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3876.458548] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 3877.959630] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 3878.625199] Lustre: Failing over lustre-MDT0001 [ 3878.633789] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.19@tcp (stopping) [ 3878.803790] Lustre: server umount lustre-MDT0001 complete [ 3892.802826] LDISKFS-fs (dm-1): recovery complete [ 3892.804660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3894.134865] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3898.812855] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1255 to 0x280000400:1281) [ 3898.812912] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1223 to 0x2c0000400:1249) [ 3900.087956] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3900.686027] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3905.600808] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3907.115537] Lustre: DEBUG MARKER: test_70b fail mds1 5 times [ 3907.780644] Lustre: Failing over lustre-MDT0000 [ 3908.192464] Lustre: server umount lustre-MDT0000 complete [ 3922.564295] LDISKFS-fs (dm-0): recovery complete [ 3922.566213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3924.027033] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3928.068758] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5011 to 0x2c0000401:5121) [ 3928.068895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5046 to 0x280000401:5153) [ 3929.245116] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3929.805438] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3933.199738] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 03:56:48 (1759823808) [ 4056.459664] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4067.241294] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 4067.894920] Lustre: Failing over lustre-MDT0000 [ 4067.908897] LustreError: 174364:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0001-osp-MDT0000: namespace resource [0x240001788:0x45:0x0].0x0 (ffff9ae048066000) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4067.926325] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 4068.978178] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 4071.391894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4071.394964] Lustre: Skipped 3 previous similar messages [ 4074.283149] Lustre: server umount lustre-MDT0000 complete [ 4088.796326] LDISKFS-fs (dm-0): recovery complete [ 4088.797754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.214944] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4095.991335] Lustre: 113317:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae172c70380 x1845304991218176/t330712502332(0) o36->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:522/0 lens 488/3152 e 0 to 0 dl 1759823982 ref 1 fl Interpret:/202/0 rc 0/0 job:'tar.0' uid:0 gid:0 projid:4294967295 [ 4096.042763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6351 to 0x2c0000401:6369) [ 4096.045901] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6384 to 0x280000401:6401) [ 4097.267637] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4097.821750] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4222.198230] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4233.027844] Lustre: DEBUG MARKER: test_70c fail mds1 2 times [ 4233.694247] Lustre: Failing over lustre-MDT0000 [ 4233.715981] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 4233.719146] Lustre: Skipped 1 previous similar message [ 4234.068851] Lustre: server umount lustre-MDT0000 complete [ 4248.681737] LDISKFS-fs (dm-0): recovery complete [ 4248.683570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.156549] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4256.636463] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7657 to 0x2c0000401:7681) [ 4256.636543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7689 to 0x280000401:7713) [ 4258.052146] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4258.603538] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4288.078745] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 04:02:43 (1759824163) [ 4411.193298] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4421.903111] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 4422.539272] Lustre: Failing over lustre-MDT0000 [ 4422.559351] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 4423.135926] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4423.140923] Lustre: Skipped 28 previous similar messages [ 4427.838876] LustreError: 178068:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4427.840794] LustreError: 178068:0:(obd_class.h:479:obd_check_dev()) Skipped 55 previous similar messages [ 4427.893964] Lustre: server umount lustre-MDT0000 complete [ 4428.256593] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.262793] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 104 previous similar messages [ 4442.136916] LDISKFS-fs (dm-0): recovery complete [ 4442.138316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4442.171357] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4442.174031] LustreError: Skipped 4 previous similar messages [ 4442.247155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4442.249147] Lustre: Skipped 6 previous similar messages [ 4442.259673] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4442.261808] Lustre: Skipped 6 previous similar messages [ 4442.738158] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4442.742353] Lustre: Skipped 6 previous similar messages [ 4443.605492] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4447.715461] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4447.717148] Lustre: Skipped 25 previous similar messages [ 4448.595403] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4448.598772] Lustre: Skipped 6 previous similar messages [ 4448.617846] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8190 to 0x2c0000401:8225) [ 4448.618425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8222 to 0x280000401:8257) [ 4449.788528] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4450.350454] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4574.402403] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4585.174274] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 4585.866266] Lustre: Failing over lustre-MDT0000 [ 4585.886902] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (stopping) [ 4585.889256] Lustre: Skipped 5 previous similar messages [ 4586.975834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4586.979439] LustreError: Skipped 8 previous similar messages [ 4592.225409] Lustre: server umount lustre-MDT0000 complete [ 4606.405380] LDISKFS-fs (dm-0): 1 orphan inode deleted [ 4606.406762] LDISKFS-fs (dm-0): recovery complete [ 4606.409851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4607.741669] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4612.798270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:8620 to 0x280000401:8641) [ 4612.798958] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:8589 to 0x2c0000401:8609) [ 4614.151990] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4614.744665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4617.905463] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 04:08:13 (1759824493) [ 4741.538943] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4752.356412] Lustre: DEBUG MARKER: test_70e fail mds1 1 times [ 4753.123350] Lustre: Failing over lustre-MDT0000 [ 4753.350440] Lustre: server umount lustre-MDT0000 complete [ 4767.272294] LDISKFS-fs (dm-0): recovery complete [ 4767.273546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4768.659379] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4775.663312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:10178 to 0x280000401:10209) [ 4775.663815] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:10147 to 0x2c0000401:10177) [ 4777.004547] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4777.681679] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4901.817412] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4912.564326] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 4913.204694] Lustre: Failing over lustre-MDT0000 [ 4913.538598] Lustre: server umount lustre-MDT0000 complete [ 4927.479672] LDISKFS-fs (dm-0): recovery complete [ 4927.481020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4928.773530] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4936.132680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12080 to 0x280000401:12097) [ 4936.145227] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12049 to 0x2c0000401:12065) [ 4937.287718] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4937.909550] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4941.432432] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 04:13:37 (1759824817) [ 4947.456926] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4949.075123] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 4949.700734] Lustre: Failing over lustre-OST0000 [ 4949.743146] Lustre: server umount lustre-OST0000 complete [ 4964.510036] LDISKFS-fs (dm-2): recovery complete [ 4964.511430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4966.413325] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4970.081834] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4970.751815] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4979.762619] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4981.273991] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 4981.951054] Lustre: Failing over lustre-OST0000 [ 4981.993888] Lustre: server umount lustre-OST0000 complete [ 4995.838974] LDISKFS-fs (dm-2): recovery complete [ 4995.840245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4997.744087] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5000.908581] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5001.484764] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5007.070023] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 04:14:42 (1759824882) [ 5129.989760] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5132.565324] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5143.404417] Lustre: DEBUG MARKER: fail mds2 mds1 1 times [ 5144.057722] Lustre: Failing over lustre-MDT0001 [ 5144.060686] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.063745] Lustre: Skipped 16 previous similar messages [ 5144.066679] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5144.070459] Lustre: Skipped 10 previous similar messages [ 5144.215432] LustreError: 189487:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 5144.217175] LustreError: 189487:0:(obd_class.h:479:obd_check_dev()) Skipped 39 previous similar messages [ 5144.323934] Lustre: server umount lustre-MDT0001 complete [ 5144.544721] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5144.548797] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 74 previous similar messages [ 5145.501133] Lustre: Failing over lustre-MDT0000 [ 5145.758116] Lustre: server umount lustre-MDT0000 complete [ 5160.533258] LDISKFS-fs (dm-1): recovery complete [ 5160.536488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5160.600503] LDISKFS-fs (dm-0): recovery complete [ 5160.601723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5160.637768] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5160.640140] LustreError: Skipped 3 previous similar messages [ 5160.641738] LustreError: 190786:0:(llog.c:1602:llog_backup()) MGC192.168.201.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5160.641966] LustreError: 190781:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.201.119@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9ae0504d3e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5160.644394] Lustre: 190786:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5160.726609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5160.728403] Lustre: Skipped 5 previous similar messages [ 5160.736446] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5160.738223] Lustre: Skipped 5 previous similar messages [ 5162.187951] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5162.283636] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5164.184883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5164.186973] Lustre: Skipped 5 previous similar messages [ 5164.960762] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5164.962390] Lustre: Skipped 19 previous similar messages [ 5165.866063] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5165.868914] Lustre: Skipped 5 previous similar messages [ 5165.882765] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4876 to 0x280000400:4897) [ 5165.882802] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4844 to 0x2c0000400:4865) [ 5165.967076] Lustre: 3627:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759825025/real 1759825025] req@ffff9ae181ae1500 x1845305005623552/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759825041 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5170.626568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12162 to 0x280000401:12193) [ 5170.627318] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12130 to 0x2c0000401:12161) [ 5171.912508] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5172.446843] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5172.963558] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5297.193653] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5299.807456] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5310.598371] Lustre: DEBUG MARKER: fail mds2 mds1 2 times [ 5311.275373] Lustre: Failing over lustre-MDT0001 [ 5311.276356] LustreError: lustre-MDT0001-osp-MDT0000: operation ldlm_enqueue to node 0@lo failed: rc = -19 [ 5311.278655] LustreError: Skipped 11 previous similar messages [ 5311.280269] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5311.524409] Lustre: server umount lustre-MDT0001 complete [ 5312.287085] Lustre: 3629:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759825035/real 1759825035] req@ffff9ae045a35f80 x1845305005624320/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759825051 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5312.294103] Lustre: 3629:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5312.743363] Lustre: Failing over lustre-MDT0000 [ 5313.025916] Lustre: server umount lustre-MDT0000 complete [ 5327.952589] LDISKFS-fs (dm-1): recovery complete [ 5327.956592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5328.065073] LDISKFS-fs (dm-0): recovery complete [ 5328.066298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5328.102128] LustreError: 194025:0:(llog.c:1602:llog_backup()) MGC192.168.201.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5328.102446] LustreError: 194026:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGC192.168.201.119@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffff9ae14a781600) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5328.104523] Lustre: 194025:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5329.247095] Lustre: 3629:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759825189/real 1759825189] req@ffff9ae052f5b100 x1845305017101952/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759825205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5329.793745] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5329.816326] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5334.389106] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4844 to 0x2c0000400:4897) [ 5334.389117] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4876 to 0x280000400:4929) [ 5334.480062] Lustre: 3629:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759825194/real 1759825194] req@ffff9ae173380a80 x1845305017102976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759825210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5334.486416] Lustre: 3629:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5339.222306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12162 to 0x280000401:12225) [ 5339.222982] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12130 to 0x2c0000401:12193) [ 5340.512581] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5341.126590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5341.688995] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5343.968086] Lustre: 3630:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759825199/real 1759825199] req@ffff9ae052f58e00 x1845305017103616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759825215 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5345.170857] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 04:20:20 (1759825220) [ 5347.756540] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5348.721312] Lustre: Failing over lustre-MDT0000 [ 5348.865427] Lustre: server umount lustre-MDT0000 complete [ 5362.807217] LDISKFS-fs (dm-0): recovery complete [ 5362.808540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.315959] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5364.195716] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5379.726039] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5379.740288] Lustre: 194033:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae170a5b480 x1845305045850752/t356482286219(356482286219) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:296/0 lens 520/3488 e 0 to 0 dl 1759825266 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5379.787574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12162 to 0x280000401:12257) [ 5379.787587] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12195 to 0x2c0000401:12225) [ 5381.044219] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5381.552558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5384.834699] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 04:21:00 (1759825260) [ 5387.264039] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5388.154175] Lustre: Failing over lustre-MDT0000 [ 5388.350679] Lustre: server umount lustre-MDT0000 complete [ 5402.298852] LDISKFS-fs (dm-0): recovery complete [ 5402.301038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5403.797243] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5404.283563] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5404.285429] LustreError: 194034:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae046380700 x1845305045850752/t356482286219(356482286219) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:321/0 lens 520/664 e 0 to 0 dl 1759825291 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5420.663842] Lustre: lustre-MDT0000: Client a3bb42a3-5116-47de-8ca5-8789ceee46eb (at 192.168.201.19@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5420.670219] Lustre: 194034:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae04922c380 x1845305045850752/t356482286219(356482286219) o101->a3bb42a3-5116-47de-8ca5-8789ceee46eb@192.168.201.19@tcp:337/0 lens 520/3488 e 0 to 0 dl 1759825307 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5420.704542] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12162 to 0x280000401:12289) [ 5420.704556] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12257) [ 5421.949897] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5422.454776] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5425.675086] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 04:21:41 (1759825301) [ 5427.055472] Lustre: Failing over lustre-OST0000 [ 5427.101800] Lustre: server umount lustre-OST0000 complete [ 5428.269866] Lustre: Failing over lustre-MDT0000 [ 5428.359532] Lustre: server umount lustre-MDT0000 complete [ 5441.106639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5442.563766] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5443.286721] Lustre: lustre-MDT0000: Denying connection for new client ca711c05-e1eb-4a36-a579-e88035b145f5 (at 192.168.201.19@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 5446.645267] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12289) [ 5450.145354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5451.637626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12162 to 0x280000401:12321) [ 5452.044277] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5455.904021] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 04:22:11 (1759825331) [ 5456.211102] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5456.213271] LustreError: 173777:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae170989c00 x1845305018553728/t377957122055(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:373/0 lens 1312/4320 e 0 to 0 dl 1759825343 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5458.545966] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5459.134982] Lustre: Failing over lustre-MDT0000 [ 5459.435721] Lustre: server umount lustre-MDT0000 complete [ 5472.735107] Lustre: 194061:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759825332/real 1759825332] req@ffff9ae170989180 x1845305018553728/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1759825348 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5472.741596] Lustre: 194061:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 5473.480363] LDISKFS-fs (dm-0): recovery complete [ 5473.482455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.870960] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5478.908598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12353) [ 5478.908621] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12321) [ 5480.048917] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5480.607432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5484.097368] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 04:22:39 (1759825359) [ 5486.733862] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5489.157776] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5489.755341] Lustre: Failing over lustre-MDT0001 [ 5489.761232] LustreError: 3628:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ae1818c3100 x1845305018585344/t0(0) o6->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-1.0' uid:0 gid:0 projid:4294967295 [ 5489.761590] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.19@tcp (stopping) [ 5489.767954] Lustre: Skipped 1 previous similar message [ 5495.862039] Lustre: server umount lustre-MDT0001 complete [ 5509.756290] LDISKFS-fs (dm-1): recovery complete [ 5509.757633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5511.109101] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5515.248366] Lustre: 194033:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae14f7eb480 x1845305060406272/t21474839143(0) o36->ca711c05-e1eb-4a36-a579-e88035b145f5@192.168.201.19@tcp:453/0 lens 560/2880 e 0 to 0 dl 1759825423 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5515.253325] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4908 to 0x2c0000400:4929) [ 5515.253335] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4940 to 0x280000400:4961) [ 5516.337764] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5516.882037] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5520.438317] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 04:23:16 (1759825396) [ 5520.765627] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5520.768048] Lustre: Skipped 1 previous similar message [ 5520.770433] LustreError: 8380:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae04f8cce00 x1845305018613760/t382252089377(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:437/0 lens 2520/4320 e 0 to 0 dl 1759825407 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5520.781317] LustreError: 8380:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 5523.070494] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5525.541706] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5526.190386] Lustre: Failing over lustre-MDT0000 [ 5526.466049] Lustre: server umount lustre-MDT0000 complete [ 5536.223160] Lustre: 204407:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759825396/real 1759825396] req@ffff9ae04f8ced80 x1845305018613760/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1759825412 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5540.351033] LDISKFS-fs (dm-0): recovery complete [ 5540.352235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.630625] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5545.999068] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12353) [ 5545.999089] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12385) [ 5547.048735] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5547.520435] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5549.435241] Lustre: Failing over lustre-MDT0001 [ 5549.529533] Lustre: server umount lustre-MDT0001 complete [ 5563.377404] LDISKFS-fs (dm-1): recovery complete [ 5563.379503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5564.715301] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5569.014077] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4972 to 0x280000400:4993) [ 5569.014125] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4940 to 0x2c0000400:4961) [ 5570.007318] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5570.496153] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5573.736378] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 04:24:09 (1759825449) [ 5579.264596] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5581.582157] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5582.175320] Lustre: Failing over lustre-MDT0000 [ 5582.476427] Lustre: server umount lustre-MDT0000 complete [ 5583.608598] LustreError: 7416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759825459 with bad export cookie 290959624605538659 [ 5583.610348] Lustre: Failing over lustre-MDT0001 [ 5583.611122] LustreError: 7416:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5583.613351] LustreError: 209705:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x20002b381:0xa:0x0].0x0 (ffff9ae17285a100) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5588.959831] LustreError: 209705:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ae175fb6d80 x1845305018673792/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5588.965176] LustreError: 209705:0:(client.c:1370:ptlrpc_import_delay_req()) Skipped 5 previous similar messages [ 5588.967328] LustreError: 209705:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5589.121374] Lustre: server umount lustre-MDT0001 complete [ 5603.545575] LDISKFS-fs (dm-0): recovery complete [ 5603.547907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.549176] LDISKFS-fs (dm-1): recovery complete [ 5603.553087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5603.630363] LustreError: 210806:0:(llog.c:1602:llog_backup()) MGC192.168.201.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5603.632747] Lustre: 210806:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5608.927473] LustreError: 3626:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9ae175fb6300 x1845305018674944/t0(0) o250->MGC192.168.201.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5610.344786] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5610.492086] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5614.141874] Lustre: 210822:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae05291ea00 x1845305060472704/t30064771140(0) o36->ca711c05-e1eb-4a36-a579-e88035b145f5@192.168.201.19@tcp:554/0 lens 560/2880 e 0 to 0 dl 1759825524 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5614.149265] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5004 to 0x280000400:5025) [ 5614.152174] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4972 to 0x2c0000400:4993) [ 5614.172431] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12385) [ 5614.172557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12417) [ 5615.189918] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5615.659114] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5616.118184] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5619.214426] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 04:24:55 (1759825495) [ 5619.527975] LustreError: 210821:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae05291fb80 x1845305060505472/t34359738436(0) o36->ca711c05-e1eb-4a36-a579-e88035b145f5@192.168.201.19@tcp:536/0 lens 560/448 e 0 to 0 dl 1759825506 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5619.535483] LustreError: 210821:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 1 previous similar message [ 5624.814603] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5625.367825] Lustre: Failing over lustre-MDT0000 [ 5625.592215] Lustre: server umount lustre-MDT0000 complete [ 5635.698684] Lustre: lustre-MDT0001: Client ca711c05-e1eb-4a36-a579-e88035b145f5 (at 192.168.201.19@tcp) reconnecting [ 5635.702773] Lustre: Skipped 2 previous similar messages [ 5635.705960] Lustre: 210822:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae046381f80 x1845305060505472/t34359738436(0) o36->ca711c05-e1eb-4a36-a579-e88035b145f5@192.168.201.19@tcp:552/0 lens 560/2880 e 0 to 0 dl 1759825522 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5639.347504] LDISKFS-fs (dm-0): recovery complete [ 5639.349479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.748235] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5644.794029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12449) [ 5644.794072] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12417) [ 5645.842107] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5646.330751] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5649.722782] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 04:25:25 (1759825525) [ 5650.046126] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5650.047539] Lustre: Skipped 2 previous similar messages [ 5652.350099] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5652.980752] Lustre: Failing over lustre-MDT0001 [ 5653.084103] Lustre: server umount lustre-MDT0001 complete [ 5666.735686] LDISKFS-fs (dm-1): recovery complete [ 5666.737765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5667.930611] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5671.917744] Lustre: 210820:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae172c68380 x1845305060535552/t34359738505(0) o36->ca711c05-e1eb-4a36-a579-e88035b145f5@192.168.201.19@tcp:588/0 lens 560/2880 e 0 to 0 dl 1759825558 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5671.922208] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5014 to 0x2c0000400:5057) [ 5671.922216] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5046 to 0x280000400:5089) [ 5672.983780] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5673.552175] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5677.180675] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 04:25:52 (1759825552) [ 5682.966773] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5685.408038] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5686.006190] Lustre: Failing over lustre-MDT0000 [ 5686.256127] Lustre: server umount lustre-MDT0000 complete [ 5700.361046] LDISKFS-fs (dm-0): recovery complete [ 5700.362344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.678600] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5705.724220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12481) [ 5705.724220] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12449) [ 5706.813575] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5707.303299] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5709.161144] Lustre: Failing over lustre-MDT0001 [ 5709.246322] Lustre: server umount lustre-MDT0001 complete [ 5723.097668] LDISKFS-fs (dm-1): recovery complete [ 5723.099818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5724.459312] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5728.244288] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5100 to 0x280000400:5121) [ 5728.244293] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5068 to 0x2c0000400:5089) [ 5729.278306] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5729.789936] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5733.249453] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 04:26:49 (1759825609) [ 5738.917585] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5741.238664] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5741.834907] Lustre: Failing over lustre-MDT0000 [ 5742.117713] Lustre: server umount lustre-MDT0000 complete [ 5743.289845] LustreError: 42252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759825619 with bad export cookie 290959624605552295 [ 5743.291383] Lustre: Failing over lustre-MDT0001 [ 5743.293680] LustreError: 42252:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5743.440413] Lustre: server umount lustre-MDT0001 complete [ 5758.398701] LDISKFS-fs (dm-0): recovery complete [ 5758.400698] LDISKFS-fs (dm-1): recovery complete [ 5758.401859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5758.403203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5758.499916] LustreError: 221379:0:(llog.c:1602:llog_backup()) MGC192.168.201.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5758.504457] Lustre: 221379:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5758.943768] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5758.944475] LustreError: 221390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.948518] Lustre: Skipped 57 previous similar messages [ 5758.953066] LustreError: 221390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 166 previous similar messages [ 5759.967124] Lustre: 3629:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759825619/real 1759825619] req@ffff9ae04f8ace00 x1845305018821248/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759825635 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5759.979169] Lustre: 3629:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5769.184075] LustreError: 3626:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9ae15fb84700 x1845305018825344/t0(0) o250->MGC192.168.201.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5769.281738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5769.285333] Lustre: Skipped 17 previous similar messages [ 5769.295568] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5769.298901] Lustre: Skipped 17 previous similar messages [ 5769.330103] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.19@tcp (not set up) [ 5769.330148] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5769.333504] Lustre: Skipped 11 previous similar messages [ 5769.339343] Lustre: Skipped 17 previous similar messages [ 5770.857482] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5771.039982] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5774.818614] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5774.821265] Lustre: Skipped 58 previous similar messages [ 5775.465186] Lustre: lustre-MDT0001: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 5775.468146] Lustre: Skipped 17 previous similar messages [ 5775.480238] Lustre: 221389:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae17098bb80 x1845305060601344/t42949673028(0) o36->ca711c05-e1eb-4a36-a579-e88035b145f5@192.168.201.19@tcp:692/0 lens 560/2880 e 0 to 0 dl 1759825662 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5775.487352] Lustre: 221389:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5775.487599] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5132 to 0x280000400:5153) [ 5775.490118] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5100 to 0x2c0000400:5121) [ 5775.500167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12513) [ 5775.500312] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12481) [ 5776.662432] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5777.168540] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5777.647420] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5780.994355] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 04:27:36 (1759825656) [ 5781.332321] LustreError: 173777:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae181907100 x1845305018851584/t403726925838(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:698/0 lens 1744/4320 e 0 to 0 dl 1759825668 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5781.343861] LustreError: 173777:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 3 previous similar messages [ 5783.707342] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5784.292960] Lustre: Failing over lustre-MDT0001 [ 5790.234664] LustreError: 223062:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5790.237567] LustreError: 223062:0:(obd_class.h:479:obd_check_dev()) Skipped 155 previous similar messages [ 5790.261606] Lustre: server umount lustre-MDT0001 complete [ 5804.021215] LDISKFS-fs (dm-1): recovery complete [ 5804.023355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5805.398139] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5809.650029] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5164 to 0x280000400:5185) [ 5809.650055] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5132 to 0x2c0000400:5153) [ 5810.773472] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5811.298627] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5814.560215] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 04:28:10 (1759825690) [ 5817.206476] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5817.806147] Lustre: Failing over lustre-MDT0000 [ 5818.116312] Lustre: server umount lustre-MDT0000 complete [ 5831.135170] Lustre: 223682:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759825690/real 1759825690] req@ffff9ae172eef480 x1845305018882176/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1728/4320 e 0 to 1 dl 1759825706 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5831.141421] Lustre: 223682:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 5831.902050] LDISKFS-fs (dm-0): recovery complete [ 5831.903191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5831.938631] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5831.941084] LustreError: Skipped 10 previous similar messages [ 5833.251086] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5837.307832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12545) [ 5837.307861] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12513) [ 5838.323845] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5838.816744] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5842.104968] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 04:28:37 (1759825717) [ 5844.870457] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5847.258522] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5847.850613] Lustre: Failing over lustre-MDT0000 [ 5848.150966] Lustre: server umount lustre-MDT0000 complete [ 5862.091530] LDISKFS-fs (dm-0): recovery complete [ 5862.092796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5863.490856] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5867.519221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12577) [ 5867.519223] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12545) [ 5868.667234] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5869.186680] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5871.039883] Lustre: Failing over lustre-MDT0001 [ 5871.117451] Lustre: server umount lustre-MDT0001 complete [ 5884.733220] LDISKFS-fs (dm-1): recovery complete [ 5884.734496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5886.006288] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5890.038695] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5164 to 0x280000400:5217) [ 5890.038713] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5132 to 0x2c0000400:5185) [ 5891.026727] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5891.491595] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5894.581286] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 04:29:30 (1759825770) [ 5897.113807] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5899.455646] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5900.055196] Lustre: Failing over lustre-MDT0000 [ 5900.223666] Lustre: server umount lustre-MDT0000 complete [ 5901.416647] LustreError: 144690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759825777 with bad export cookie 290959624605562669 [ 5901.418129] Lustre: Failing over lustre-MDT0001 [ 5901.420034] LustreError: 144690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5901.423089] LustreError: 230849:0:(ldlm_resource.c:981:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x20002b381:0x1f:0x0].0x0 (ffff9ae16103c800) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 5906.911856] LustreError: 230849:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9ae049b1ce00 x1845305018959616/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5906.916764] LustreError: 230849:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5907.059256] Lustre: server umount lustre-MDT0001 complete [ 5921.868366] LDISKFS-fs (dm-1): recovery complete [ 5921.868397] LDISKFS-fs (dm-0): recovery complete [ 5921.869803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5921.870922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.948437] LustreError: 231949:0:(llog.c:1602:llog_backup()) MGC192.168.201.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5921.950961] Lustre: 231949:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5926.879579] LustreError: 3626:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9ae050c03b80 x1845305018960768/t0(0) o250->MGC192.168.201.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5927.021209] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 5927.023354] LustreError: Skipped 15 previous similar messages [ 5928.503725] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5928.610101] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5933.248104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12609) [ 5933.248115] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12577) [ 5936.318439] Lustre: 232709:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae04eaf3480 x1845305060677376/t55834574850(0) o36->ca711c05-e1eb-4a36-a579-e88035b145f5@192.168.201.19@tcp:124/0 lens 496/2888 e 0 to 0 dl 1759825849 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 5936.324011] Lustre: 232709:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5936.326867] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5164 to 0x280000400:5249) [ 5936.326916] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5132 to 0x2c0000400:5217) [ 5937.369965] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5937.853029] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5938.333859] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5941.593516] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 04:30:17 (1759825817) [ 5941.931298] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5941.932589] Lustre: Skipped 6 previous similar messages [ 5944.546950] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5945.158795] Lustre: Failing over lustre-MDT0000 [ 5945.338111] Lustre: server umount lustre-MDT0000 complete [ 5959.101308] LDISKFS-fs (dm-0): recovery complete [ 5959.102518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5960.442579] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5964.292963] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12609) [ 5964.292992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12641) [ 5965.394671] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5965.907104] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5997.170449] Lustre: lustre-MDT0001: Client ca711c05-e1eb-4a36-a579-e88035b145f5 (at 192.168.201.19@tcp) reconnecting [ 5997.172650] Lustre: Skipped 1 previous similar message [ 5999.264614] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 04:31:15 (1759825875) [ 6001.908962] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6002.471855] Lustre: Failing over lustre-MDT0001 [ 6002.549965] Lustre: server umount lustre-MDT0001 complete [ 6016.311810] LDISKFS-fs (dm-1): recovery complete [ 6016.313228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6017.669509] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6021.620727] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5164 to 0x280000400:5281) [ 6021.620738] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5132 to 0x2c0000400:5249) [ 6022.728201] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6023.227717] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6026.509051] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 04:31:42 (1759825902) [ 6029.101777] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6031.294609] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6031.828778] Lustre: Failing over lustre-MDT0000 [ 6031.988289] Lustre: server umount lustre-MDT0000 complete [ 6045.822414] LDISKFS-fs (dm-0): recovery complete [ 6045.823647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.145211] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6051.328984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12673) [ 6051.329108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12641) [ 6052.330820] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6052.804703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6054.635119] Lustre: Failing over lustre-MDT0001 [ 6054.715503] Lustre: server umount lustre-MDT0001 complete [ 6068.392190] LDISKFS-fs (dm-1): recovery complete [ 6068.393485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6069.673834] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6073.843626] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5164 to 0x280000400:5313) [ 6073.843626] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5132 to 0x2c0000400:5281) [ 6074.933326] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6075.439441] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6078.724350] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 04:32:34 (1759825954) [ 6079.093529] LustreError: 231959:0:(ldlm_lib.c:3324:target_send_reply_msg()) @@@ dropping reply req@ffff9ae1755bed80 x1845305060734848/t68719476738(0) o36->ca711c05-e1eb-4a36-a579-e88035b145f5@192.168.201.19@tcp:240/0 lens 496/456 e 0 to 0 dl 1759825965 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6079.099746] LustreError: 231959:0:(ldlm_lib.c:3324:target_send_reply_msg()) Skipped 6 previous similar messages [ 6081.496554] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6083.790969] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6084.355980] Lustre: Failing over lustre-MDT0000 [ 6084.476042] Lustre: server umount lustre-MDT0000 complete [ 6085.663953] LustreError: 6474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759825961 with bad export cookie 290959624605570166 [ 6085.665461] Lustre: Failing over lustre-MDT0001 [ 6085.667732] LustreError: 6474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6085.814688] Lustre: server umount lustre-MDT0001 complete [ 6100.637070] LDISKFS-fs (dm-0): recovery complete [ 6100.639412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6100.640270] LDISKFS-fs (dm-1): recovery complete [ 6100.645643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6100.733197] LustreError: 242588:0:(llog.c:1602:llog_backup()) MGC192.168.201.119@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6100.735347] Lustre: 242588:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.119@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6104.543089] Lustre: 3630:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759825964/real 1759825964] req@ffff9ae15fb85f80 x1845305019102208/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1759825980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6104.551558] Lustre: 3630:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6111.200023] LustreError: 3626:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9ae173b99c00 x1845305019106304/t0(0) o250->MGC192.168.201.119@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6112.786934] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6112.911832] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6117.425800] Lustre: 242604:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9ae173b9aa00 x1845305060734848/t68719476738(0) o36->ca711c05-e1eb-4a36-a579-e88035b145f5@192.168.201.19@tcp:279/0 lens 496/2888 e 0 to 0 dl 1759826004 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6117.430074] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:5132 to 0x2c0000400:5313) [ 6117.430075] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5164 to 0x280000400:5345) [ 6117.437173] Lustre: 242604:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 3 previous similar messages [ 6117.452909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12323 to 0x280000401:12705) [ 6117.452918] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12227 to 0x2c0000401:12673) [ 6118.512617] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6119.035327] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6119.537064] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6122.882752] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 04:33:18 (1759825998) [ 6123.452536] Lustre: 243822:0:(genops.c:1791:obd_export_evict_by_uuid()) lustre-MDT0000: evicting ca711c05-e1eb-4a36-a579-e88035b145f5 at adminstrative request [ 6127.927847] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 04:33:23 (1759826003) [ 6129.628040] Lustre: Failing over lustre-MDT0000 [ 6129.830069] Lustre: server umount lustre-MDT0000 complete [ 6142.546721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.875650] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6148.105940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12756 to 0x280000401:12801) [ 6148.105970] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12725 to 0x2c0000401:12769) [ 6149.059745] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6149.515364] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6152.575106] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 04:33:48 (1759826028) [ 6155.685850] Lustre: Failing over lustre-OST0000 [ 6155.753577] Lustre: server umount lustre-OST0000 complete [ 6168.333207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6170.141921] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6173.116375] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6173.638757] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6177.210878] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 04:34:12 (1759826052) [ 6178.468083] Lustre: Failing over lustre-MDT0000 [ 6178.593588] Lustre: server umount lustre-MDT0000 complete [ 6180.880534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.245519] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6182.910958] Lustre: lustre-MDT0000: Denying connection for new client 1668e7d4-7026-4731-983a-b977976ac284 (at 192.168.201.19@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 6182.915596] Lustre: Skipped 1 previous similar message [ 6186.484291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12756 to 0x280000401:12833) [ 6186.484317] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12725 to 0x2c0000401:12801) [ 6190.049377] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 04:34:25 (1759826065) [ 6192.667326] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6193.463759] Lustre: Failing over lustre-OST0000 [ 6193.492595] Lustre: server umount lustre-OST0000 complete [ 6207.181985] LDISKFS-fs (dm-2): recovery complete [ 6207.183470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6208.993173] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6211.900211] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6212.405879] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6215.567648] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 04:34:51 (1759826091) [ 6218.074497] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6219.868053] Lustre: Failing over lustre-OST0000 [ 6219.901879] Lustre: server umount lustre-OST0000 complete [ 6233.456376] LDISKFS-fs (dm-2): recovery complete [ 6233.457645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6235.332216] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.19@tcp inode [0x2000320e1:0x5:0x0] object 0x280000401:12835 extent [0-4194303]: client csum 48651e0, server csum b5a41a4d [ 6235.417801] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6238.897465] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6239.787985] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6244.598972] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 04:35:20 (1759826120) [ 6248.366514] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6251.965240] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6254.909153] Lustre: Failing over lustre-MDT0000 [ 6255.110234] Lustre: server umount lustre-MDT0000 complete [ 6256.859805] LustreError: 6474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759826132 with bad export cookie 290959624605596934 [ 6256.863923] Lustre: Failing over lustre-OST0000 [ 6256.865302] LustreError: 6474:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6258.959269] Lustre: server umount lustre-OST0000 complete [ 6281.368834] LDISKFS-fs (dm-0): recovery complete [ 6281.371971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6282.736226] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x409b23cac990bdf [ 6286.906849] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6289.172027] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12725 to 0x2c0000401:12833) [ 6305.044791] LDISKFS-fs (dm-2): recovery complete [ 6305.048256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6307.610725] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6314.318507] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 04:36:30 (1759826190) [ 6319.565551] Lustre: Failing over lustre-OST0000 [ 6319.625290] Lustre: server umount lustre-OST0000 complete [ 6321.201467] Lustre: Failing over lustre-MDT0000 [ 6321.424105] Lustre: server umount lustre-MDT0000 complete [ 6334.752458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.662751] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6340.097356] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12725 to 0x2c0000401:12865) [ 6344.495586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6347.134289] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6348.167362] Lustre: lustre-OST0000: Denying connection for new client 372a80f4-ac85-4b89-9dfa-371d9ceccc13 (at 192.168.201.19@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 6348.177374] Lustre: Skipped 1 previous similar message [ 6353.522729] Lustre: lustre-OST0000: Denying connection for new client 372a80f4-ac85-4b89-9dfa-371d9ceccc13 (at 192.168.201.19@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:01 [ 6363.762451] Lustre: lustre-OST0000: Denying connection for new client 372a80f4-ac85-4b89-9dfa-371d9ceccc13 (at 192.168.201.19@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:51 [ 6363.768023] Lustre: Skipped 1 previous similar message [ 6384.242549] Lustre: lustre-OST0000: Denying connection for new client 372a80f4-ac85-4b89-9dfa-371d9ceccc13 (at 192.168.201.19@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:30 [ 6384.250451] Lustre: Skipped 3 previous similar messages [ 6415.000193] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6415.002837] Lustre: 256860:0:(genops.c:1618:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 1668e7d4-7026-4731-983a-b977976ac284@ [ 6415.006738] Lustre: 256860:0:(genops.c:1618:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6415.010421] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6415.018078] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6415.018150] Lustre: lustre-OST0000: Recovery over after 1:10, of 3 clients 2 recovered and 1 was evicted. [ 6415.020193] Lustre: Skipped 68 previous similar messages [ 6415.023655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12884 to 0x280000401:12905) [ 6415.028383] Lustre: Skipped 21 previous similar messages [ 6421.535189] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 6426.729356] Lustre: DEBUG MARKER: free_before: 7646296 free_after: 7646296 [ 6429.653657] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 04:38:25 (1759826305) [ 6431.731632] Lustre: Failing over lustre-OST0000 [ 6431.757066] LustreError: 258098:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6431.761477] LustreError: 258098:0:(obd_class.h:479:obd_check_dev()) Skipped 147 previous similar messages [ 6431.782386] Lustre: server umount lustre-OST0000 complete [ 6432.738931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6432.745589] Lustre: Skipped 61 previous similar messages [ 6432.749246] LustreError: 42257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6432.755743] LustreError: 42257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 256 previous similar messages [ 6446.498236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6446.612119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6446.618732] Lustre: Skipped 22 previous similar messages [ 6446.627621] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6446.630153] Lustre: Skipped 22 previous similar messages [ 6447.924879] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6447.929380] Lustre: Skipped 22 previous similar messages [ 6449.510680] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6455.168621] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 04:38:50 (1759826330) [ 6457.457160] Lustre: Failing over lustre-OST0000 [ 6457.517163] Lustre: server umount lustre-OST0000 complete [ 6471.925264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6473.215260] LustreError: 259843:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6473.220374] LustreError: 259843:0:(ldlm_lib.c:2883:target_recovery_thread()) Skipped 78 previous similar messages [ 6474.942651] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6479.327479] Lustre: *** cfs_fail_loc=715, val=40*** [ 6489.567490] Lustre: 3626:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1759826349/real 1759826349] req@ffff9ae172c68e00 x1845305019358592/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1759826365 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 6489.582946] Lustre: 3626:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [ 6489.588052] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [ 6489.592635] Lustre: Skipped 1 previous similar message [ 6495.711203] Lustre: *** cfs_fail_loc=715, val=40*** [ 6495.712680] Lustre: Skipped 2 previous similar messages [ 6505.074637] Lustre: lustre-OST0000: Client 372a80f4-ac85-4b89-9dfa-371d9ceccc13 (at 192.168.201.19@tcp) reconnected, waiting for 3 clients in recovery for 1:08 [ 6505.080908] Lustre: Skipped 1 previous similar message [ 6511.584287] Lustre: *** cfs_fail_loc=715, val=40*** [ 6511.587133] Lustre: Skipped 2 previous similar messages [ 6513.271155] LustreError: 259843:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6513.274476] LustreError: 259843:0:(ldlm_lib.c:2883:target_recovery_thread()) Skipped 78 previous similar messages [ 6515.230477] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6515.949100] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6520.135136] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 04:39:55 (1759826395) [ 6522.041140] Lustre: Failing over lustre-MDT0000 [ 6522.351977] Lustre: server umount lustre-MDT0000 complete [ 6536.588686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6536.679742] LustreError: MGC192.168.201.119@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6536.686212] LustreError: Skipped 9 previous similar messages [ 6538.919258] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6542.318083] LustreError: 261286:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6547.423163] Lustre: *** cfs_fail_loc=715, val=80*** [ 6547.425211] Lustre: Skipped 2 previous similar messages [ 6557.298128] Lustre: lustre-MDT0000: Client 372a80f4-ac85-4b89-9dfa-371d9ceccc13 (at 192.168.201.19@tcp) reconnected, waiting for 2 clients in recovery for 1:25 [ 6557.304956] Lustre: Skipped 2 previous similar messages [ 6558.687659] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6563.808210] Lustre: *** cfs_fail_loc=715, val=80*** [ 6563.810345] Lustre: Skipped 1 previous similar message [ 6573.682732] Lustre: lustre-MDT0000: Client 372a80f4-ac85-4b89-9dfa-371d9ceccc13 (at 192.168.201.19@tcp) reconnected, waiting for 2 clients in recovery for 1:09 [ 6575.074734] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6580.191223] Lustre: *** cfs_fail_loc=715, val=80*** [ 6580.194036] Lustre: Skipped 1 previous similar message [ 6589.042963] Lustre: lustre-MDT0000: Client 372a80f4-ac85-4b89-9dfa-371d9ceccc13 (at 192.168.201.19@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 6590.432430] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6596.575234] Lustre: *** cfs_fail_loc=715, val=80*** [ 6596.577491] Lustre: Skipped 2 previous similar messages [ 6605.426513] Lustre: lustre-MDT0000: Client 372a80f4-ac85-4b89-9dfa-371d9ceccc13 (at 192.168.201.19@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [ 6606.816305] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6621.810366] Lustre: lustre-MDT0000: Client 372a80f4-ac85-4b89-9dfa-371d9ceccc13 (at 192.168.201.19@tcp) reconnected, waiting for 2 clients in recovery for 0:21 [ 6622.415119] LustreError: 261286:0:(ldlm_lib.c:2883:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6622.425546] Lustre: 261286:0:(ldlm_lib.c:2929:target_recovery_thread()) too long recovery - read logs [ 6622.431109] LustreError: dumping log to /tmp/lustre-log.1759826498.261286 [ 6622.545966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12919 to 0x280000401:12937) [ 6622.546776] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12878 to 0x2c0000401:12897) [ 6624.763717] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6625.694758] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6630.987675] Lustre: DEBUG MARKER: == replay-single test complete, duration 6478 sec ======== 04:41:46 (1759826506) [ 6631.783551] Lustre: DEBUG MARKER: === replay-single: start cleanup 04:41:47 (1759826507) === [ 6637.557530] Lustre: DEBUG MARKER: === replay-single: finish cleanup 04:41:53 (1759826513) === [ 6668.258431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6668.262331] Lustre: Skipped 22 previous similar messages [ 6674.101726] Lustre: server umount lustre-MDT0000 complete [ 6678.433192] LustreError: 6475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1759826554 with bad export cookie 290959624605609576 [ 6678.439310] LustreError: 6475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6678.658400] Lustre: server umount lustre-MDT0001 complete [ 6684.929251] Lustre: server umount lustre-OST0000 complete [ 6689.220343] Lustre: server umount lustre-OST0001 complete [ 6697.822226] Lustre: DEBUG MARKER: oleg119-server.virtnet: executing unload_modules_local [ 6699.406616] Key type lgssc unregistered [ 6699.578465] LNet: 264446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6699.582452] LNetError: 264446:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6699.592312] LNet: Removed LNI 192.168.201.119@tcp [ 6700.050139] Key type .llcrypt unregistered [ 6700.051546] Key type ._llcrypt unregistered