[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 396473076 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002243] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006416] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007007] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008004] pid_max: default: 32768 minimum: 301 [ 0.009063] LSM: Security Framework initializing [ 0.010024] Yama: becoming mindful. [ 0.011018] SELinux: Initializing. [ 0.011812] *** VALIDATE selinux *** [ 0.018490] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022149] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024005] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025060] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026046] *** VALIDATE tmpfs *** [ 0.027309] *** VALIDATE proc *** [ 0.028135] *** VALIDATE cgroup *** [ 0.029004] *** VALIDATE cgroup2 *** [ 0.030167] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031099] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033028] Spectre V2 : User space: Vulnerable [ 0.034003] Speculative Store Bypass: Vulnerable [ 0.036895] debug: unmapping init [mem 0xffffffff85059000-0xffffffff85060fff] [ 0.038116] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039381] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040010] ... version: 2 [ 0.040900] ... bit width: 48 [ 0.041005] ... generic registers: 4 [ 0.041867] ... value mask: 0000ffffffffffff [ 0.042005] ... max period: 00007fffffffffff [ 0.043005] ... fixed-purpose events: 3 [ 0.043881] ... event mask: 000000070000000f [ 0.044114] rcu: Hierarchical SRCU implementation. [ 0.045982] smp: Bringing up secondary CPUs ... [ 0.046334] x86: Booting SMP configuration: [ 0.047010] .... node #0, CPUs: #1 #2 #3 [ 0.049432] smp: Brought up 1 node, 4 CPUs [ 0.050863] smpboot: Max logical packages: 1 [ 0.051007] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.203205] node 0 deferred pages initialised in 151ms [ 0.205417] devtmpfs: initialized [ 0.206170] x86/mm: Memory block size: 128MB [ 0.208201] gcov: version magic: 0x41383552 [ 0.209543] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.210055] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.211151] pinctrl core: initialized pinctrl subsystem [ 0.212093] [ 0.212457] ************************************************************* [ 0.213007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.214006] ** ** [ 0.215006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.216006] ** ** [ 0.217005] ** This means that this kernel is built to expose internal ** [ 0.218006] ** IOMMU data structures, which may compromise security on ** [ 0.219006] ** your system. ** [ 0.220006] ** ** [ 0.221006] ** If you see this message and you are not debugging the ** [ 0.222005] ** kernel, report this immediately to your vendor! ** [ 0.223006] ** ** [ 0.224005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.225006] ************************************************************* [ 0.226407] NET: Registered protocol family 16 [ 0.227280] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.228026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.229038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.230239] cpuidle: using governor menu [ 0.231551] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.234415] PCI: Using configuration type 1 for base access [ 0.236100] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.243088] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.244018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.246113] cryptd: max_cpu_qlen set to 1000 [ 0.247176] ACPI: Added _OSI(Module Device) [ 0.248008] ACPI: Added _OSI(Processor Device) [ 0.248849] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.253097] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.257407] ACPI: Interpreter enabled [ 0.259045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260008] ACPI: Using IOAPIC for interrupt routing [ 0.262042] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.264267] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.272504] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.274023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.277016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.280038] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283949] acpiphp: Slot [2] registered [ 0.286083] acpiphp: Slot [5] registered [ 0.287079] acpiphp: Slot [6] registered [ 0.288058] acpiphp: Slot [7] registered [ 0.289055] acpiphp: Slot [8] registered [ 0.291079] acpiphp: Slot [9] registered [ 0.292057] acpiphp: Slot [10] registered [ 0.293055] acpiphp: Slot [3] registered [ 0.294050] acpiphp: Slot [4] registered [ 0.296063] acpiphp: Slot [11] registered [ 0.297056] acpiphp: Slot [12] registered [ 0.298062] acpiphp: Slot [13] registered [ 0.300052] acpiphp: Slot [14] registered [ 0.301050] acpiphp: Slot [15] registered [ 0.302050] acpiphp: Slot [16] registered [ 0.303050] acpiphp: Slot [17] registered [ 0.305049] acpiphp: Slot [18] registered [ 0.306060] acpiphp: Slot [19] registered [ 0.307063] acpiphp: Slot [20] registered [ 0.309051] acpiphp: Slot [21] registered [ 0.310048] acpiphp: Slot [22] registered [ 0.311050] acpiphp: Slot [23] registered [ 0.313073] acpiphp: Slot [24] registered [ 0.314067] acpiphp: Slot [25] registered [ 0.316086] acpiphp: Slot [26] registered [ 0.317054] acpiphp: Slot [27] registered [ 0.319079] acpiphp: Slot [28] registered [ 0.320054] acpiphp: Slot [29] registered [ 0.321052] acpiphp: Slot [30] registered [ 0.322045] acpiphp: Slot [31] registered [ 0.323045] PCI host bridge to bus 0000:00 [ 0.323939] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325008] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327011] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328007] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.329010] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.331009] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.332096] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.333688] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.337376] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.343642] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.346667] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348007] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350007] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.352009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.354349] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.355648] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.358019] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.359622] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362008] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.372007] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379007] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.381980] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.385783] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.397754] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.402872] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.407006] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.411008] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.424950] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.431413] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.435795] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.439010] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.452920] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.459383] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.461798] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.466777] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.478007] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.483779] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.486885] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.490713] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.503009] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.507925] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.510745] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.513773] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.527018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.533237] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.535233] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.537233] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.540246] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.541121] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.545167] iommu: Default domain type: Passthrough [ 0.547405] SCSI subsystem initialized [ 0.548077] ACPI: bus type USB registered [ 0.550074] usbcore: registered new interface driver usbfs [ 0.552039] usbcore: registered new interface driver hub [ 0.553042] usbcore: registered new device driver usb [ 0.554095] pps_core: LinuxPPS API ver. 1 registered [ 0.555007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.557022] PTP clock support registered [ 0.558120] EDAC MC: Ver: 3.0.0 [ 0.559292] PCI: Using ACPI for IRQ routing [ 0.560441] NetLabel: Initializing [ 0.561009] NetLabel: domain hash size = 128 [ 0.563012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.564047] NetLabel: unlabeled traffic allowed by default [ 0.566078] vgaarb: loaded [ 0.568141] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.569006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.574289] clocksource: Switched to clocksource kvm-clock [ 0.652828] VFS: Disk quotas dquot_6.6.0 [ 0.653757] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.655197] *** VALIDATE ramfs *** [ 0.655880] *** VALIDATE hugetlbfs *** [ 0.657373] pnp: PnP ACPI init [ 0.658959] pnp: PnP ACPI: found 6 devices [ 0.670900] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.672860] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.674150] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.675422] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.676893] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.678197] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.679888] NET: Registered protocol family 2 [ 0.681361] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.684651] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.686706] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.690452] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.692502] TCP: Hash tables configured (established 65536 bind 65536) [ 0.694159] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.696202] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.698764] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.701375] NET: Registered protocol family 1 [ 0.703987] RPC: Registered named UNIX socket transport module. [ 0.706054] RPC: Registered udp transport module. [ 0.707607] RPC: Registered tcp transport module. [ 0.709087] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.711632] NET: Registered protocol family 44 [ 0.712624] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.713830] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.715122] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.716560] PCI: CLS 0 bytes, default 64 [ 0.717556] Unpacking initramfs... [ 1.854639] debug: unmapping init [mem 0xffff9f06fcc54000-0xffff9f06fffbffff] [ 1.857276] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.858662] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.860513] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.274204] Initialise system trusted keyrings [ 2.275328] Key type blacklist registered [ 2.276447] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.282419] zbud: loaded [ 2.284296] *** VALIDATE nfs *** [ 2.285062] *** VALIDATE nfs4 *** [ 2.285968] pstore: using deflate compression [ 2.287605] Platform Keyring initialized [ 2.364202] NET: Registered protocol family 38 [ 2.365300] Key type asymmetric registered [ 2.366184] Asymmetric key parser 'x509' registered [ 2.367268] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.369034] io scheduler mq-deadline registered [ 2.370059] io scheduler kyber registered [ 2.370929] io scheduler bfq registered [ 2.372026] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.374336] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.376206] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.378039] ACPI: Power Button [PWRF] [ 2.382201] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.386850] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.395025] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.400566] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.409484] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.435181] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.461342] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.464583] Non-volatile memory driver v1.3 [ 2.466020] Linux agpgart interface v0.103 [ 2.489714] virtio_blk virtio1: [vda] 134144 512-byte logical blocks (68.7 MB/65.5 MiB) [ 2.492492] vda: detected capacity change from 0 to 68681728 [ 2.502957] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.504691] vdb: detected capacity change from 0 to 1073741824 [ 2.513692] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.515375] vdc: detected capacity change from 0 to 2621440000 [ 2.524853] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.526563] vdd: detected capacity change from 0 to 2621440000 [ 2.535119] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.536886] vde: detected capacity change from 0 to 4294967296 [ 2.546355] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.548102] vdf: detected capacity change from 0 to 4294967296 [ 2.551974] libphy: Fixed MDIO Bus: probed [ 2.555551] usbcore: registered new interface driver usbserial_generic [ 2.557125] usbserial: USB Serial support registered for generic [ 2.558761] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.561728] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.562991] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.564590] mousedev: PS/2 mouse device common for all mice [ 2.566431] rtc_cmos 00:05: RTC can wake from S4 [ 2.568326] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.569664] rtc_cmos 00:05: registered as rtc0 [ 2.572465] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.574260] intel_pstate: CPU model not supported [ 2.575996] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.576399] hid: raw HID events driver (C) Jiri Kosina [ 2.579167] usbcore: registered new interface driver usbhid [ 2.580481] usbhid: USB HID core driver [ 2.581543] drop_monitor: Initializing network drop monitor service [ 2.581594] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.583058] Initializing XFRM netlink socket [ 2.586209] NET: Registered protocol family 10 [ 2.588264] Segment Routing with IPv6 [ 2.589181] NET: Registered protocol family 17 [ 2.590417] mpls_gso: MPLS GSO support [ 2.594749] RAS: Correctable Errors collector initialized. [ 2.596179] AVX version of gcm_enc/dec engaged. [ 2.597290] AES CTR mode by8 optimization enabled [ 2.638795] sched_clock: Marking stable (2638773255, 0)->(3269931180, -631157925) [ 2.640750] registered taskstats version 1 [ 2.642106] Loading compiled-in X.509 certificates [ 2.643372] zswap: loaded using pool lzo/zbud [ 2.656533] Key type big_key registered [ 2.663665] Key type encrypted registered [ 2.664699] ima: No TPM chip found, activating TPM-bypass! [ 2.665870] ima: Allocated hash algorithm: sha1 [ 2.666850] ima: No architecture policies found [ 2.667858] evm: Initialising EVM extended attributes: [ 2.668948] evm: security.selinux [ 2.669637] evm: security.ima [ 2.670221] evm: security.capability [ 2.670954] evm: HMAC attrs: 0x1 [ 2.672426] rtc_cmos 00:05: setting system clock to 2026-05-09 21:03:42 UTC (1778360622) [ 2.676317] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 2.678148] debug: unmapping init [mem 0xffffffff84d82000-0xffffffff85058fff] [ 2.684053] Write protecting the kernel read-only data: 28672k [ 2.686034] debug: unmapping init [mem 0xffffffff83403000-0xffffffff835fffff] [ 2.687779] debug: unmapping init [mem 0xffffffff83d14000-0xffffffff83dfffff] [ 2.708914] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.713974] systemd[1]: Detected virtualization kvm. [ 2.715146] systemd[1]: Detected architecture x86-64. [ 2.716278] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.736837] systemd[1]: No hostname configured. [ 2.738117] systemd[1]: Set hostname to . [ 2.739448] random: systemd: uninitialized urandom read (16 bytes read) [ 2.741140] systemd[1]: Initializing machine ID from random generator. [ 2.777698] random: ln: uninitialized urandom read (6 bytes read) [ 2.842815] random: systemd: uninitialized urandom read (16 bytes read) [ 2.844453] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 2.847188] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 2.849862] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ 3.009739] random: fast init done [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.259297] device-mapper: uevent: version 1.0.3 [ 3.260526] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 3.820772] virtio_net virtio0 ens2: renamed from eth0 [ 3.945513] scsi host0: ata_piix [ 3.968461] scsi host1: ata_piix [ 3.969568] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 3.971067] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 7.322803] dracut-initqueue[581]: RTNETLINK answers: File exists [ 8.174414] random: crng init done [ 8.175257] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 8.462194] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 9.290792] printk: systemd: 25 output lines suppressed due to ratelimiting [ 9.438185] SELinux: Disabled at runtime. [ 9.485711] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.490349] systemd[1]: Detected virtualization kvm. [ 9.491335] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 9.771287] systemd[1]: initrd-switch-root.service: Succeeded. [ 9.773017] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 9.775969] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 9.778146] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 9.779943] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 9.784401] systemd[1]: Starting Journal Service... Starting Journal Service... [ 9.790918] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... Mountin[ 9.822583] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 10.159860] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 10.445337] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 10.472862] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 10.554122] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 10.572904] EDAC sbridge: Ver: 1.1.2 [ 11.581817] Key type dns_resolver registered [ 11.859467] NFS: Registering the id_resolver key type [ 11.861438] Key type id_resolver registered [ 11.863009] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg432-server login: [ 33.958665] libcfs: loading out-of-tree module taints kernel. [ 33.974875] Key type ._llcrypt registered [ 33.976441] Key type .llcrypt registered [ 34.023416] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 41.443552] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 41.980698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 41.987016] alg: No test for adler32 (adler32-zlib) [ 42.956897] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 43.220605] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 44.815167] Key type lgssc registered [ 45.262421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 52.107252] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 69.875065] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 75.379090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 75.391461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 76.534220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 76.556587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 76.614753] Lustre: lustre-MDT0000: new disk, initializing [ 76.660490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 76.676352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 78.565484] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 85.478386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 85.540227] Lustre: 6476:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 85.562995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 85.567189] Lustre: Skipped 1 previous similar message [ 85.620121] Lustre: lustre-MDT0001: new disk, initializing [ 85.649942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 85.663307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 85.669853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 87.437190] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 90.331280] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 94.528526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 94.673366] Lustre: lustre-OST0000: new disk, initializing [ 94.676844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 94.701476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 97.193570] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 100.853357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 100.859305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 100.895030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 104.463958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 104.540703] Lustre: lustre-OST0001: new disk, initializing [ 104.544437] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 104.580527] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 107.439199] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.574260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 110.579295] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 110.597348] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 114.506567] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 120.940257] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 128.566399] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing check_logdir /tmp/testlogs/ [ 130.411166] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing yml_node [ 132.584332] Lustre: DEBUG MARKER: Client: 2.17.52.4 [ 133.844027] Lustre: DEBUG MARKER: MDS: 2.17.52.4 [ 135.133192] Lustre: DEBUG MARKER: OSS: 2.17.52.4 [ 135.930391] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Sat May 9 17:05:55 EDT 2026 [ 144.263457] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 145.228437] Lustre: DEBUG MARKER: === replay-single: start setup 17:06:04 (1778360764) === [ 147.068717] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing check_config_client /mnt/lustre [ 155.710591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 156.972339] Lustre: 13145:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 158.597131] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.238433] Lustre: DEBUG MARKER: === replay-single: finish setup 17:06:20 (1778360780) === [ 162.165831] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 17:06:21 (1778360781) [ 165.756473] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 166.729505] Lustre: Failing over lustre-MDT0000 [ 166.880100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 166.885276] Lustre: Skipped 3 previous similar messages [ 166.893470] Lustre: server umount lustre-MDT0000 complete [ 172.000089] LustreError: 7414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 172.010219] LustreError: 7414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 173.197689] LustreError: 9449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 177.120207] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 177.130919] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 182.240379] LustreError: 7414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 182.250895] LustreError: 7414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 182.831049] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 182.833467] LDISKFS-fs (dm-0): recovery complete [ 182.842096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.862939] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 182.951239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.438947] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 184.630420] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.386655] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 188.406666] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 190.080101] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 190.863656] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 195.595790] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 17:06:55 (1778360815) [ 196.661788] Lustre: Failing over lustre-OST0000 [ 196.719973] Lustre: server umount lustre-OST0000 complete [ 198.111882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 198.117474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 198.127105] LustreError: 9029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 198.625534] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 208.864837] LustreError: 8368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 208.873493] LustreError: 8368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 210.515700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.600347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.892173] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 211.940355] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 211.940371] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 211.948949] Lustre: Skipped 3 previous similar messages [ 213.113567] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.752348] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 217.588244] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 222.205321] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 17:07:22 (1778360842) [ 225.780071] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 226.746092] Lustre: Failing over lustre-MDT0000 [ 226.839742] Lustre: server umount lustre-MDT0000 complete [ 229.343913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 229.349305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 229.357244] Lustre: Skipped 1 previous similar message [ 229.360534] LustreError: 7414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 229.369417] LustreError: 7414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 231.392739] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 231.400976] Lustre: Skipped 2 previous similar messages [ 242.441845] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 242.444715] LDISKFS-fs (dm-0): recovery complete [ 242.449760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.509683] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 242.614679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.338988] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.384782] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 245.390113] Lustre: lustre-MDT0000: Denying connection for new client 34322886-4cb2-4120-a5d6-83847305eedf (at 192.168.204.32@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 247.784253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 247.790221] Lustre: Skipped 1 previous similar message [ 250.509340] Lustre: lustre-MDT0000: Denying connection for new client 34322886-4cb2-4120-a5d6-83847305eedf (at 192.168.204.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 250.524575] Lustre: Skipped 1 previous similar message [ 255.629395] Lustre: lustre-MDT0000: Denying connection for new client 34322886-4cb2-4120-a5d6-83847305eedf (at 192.168.204.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 260.749410] Lustre: lustre-MDT0000: Denying connection for new client 34322886-4cb2-4120-a5d6-83847305eedf (at 192.168.204.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 265.869345] Lustre: lustre-MDT0000: Denying connection for new client 34322886-4cb2-4120-a5d6-83847305eedf (at 192.168.204.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 276.109235] Lustre: lustre-MDT0000: Denying connection for new client 34322886-4cb2-4120-a5d6-83847305eedf (at 192.168.204.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 276.120118] Lustre: Skipped 1 previous similar message [ 296.588894] Lustre: lustre-MDT0000: Denying connection for new client 34322886-4cb2-4120-a5d6-83847305eedf (at 192.168.204.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 296.598901] Lustre: Skipped 3 previous similar messages [ 315.500164] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 315.504259] Lustre: 17804:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 0a6b588c-34cc-48af-beff-4c7b7c57de1d@ [ 315.512038] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 315.519272] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 315.519450] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 315.528700] Lustre: Skipped 2 previous similar messages [ 315.547723] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 315.547749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 320.804494] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 17:09:00 (1778360940) [ 324.089974] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 324.936668] Lustre: Failing over lustre-MDT0000 [ 325.051508] Lustre: server umount lustre-MDT0000 complete [ 326.111656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 326.116466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.124268] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.134234] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 340.199406] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 340.202187] LDISKFS-fs (dm-0): recovery complete [ 340.207271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 340.261281] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.348067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 341.908961] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 343.042289] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 343.047627] Lustre: lustre-MDT0000: Denying connection for new client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 343.057701] Lustre: Skipped 3 previous similar messages [ 345.570758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 409.741137] Lustre: lustre-MDT0000: Denying connection for new client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 409.752143] Lustre: Skipped 13 previous similar messages [ 413.500166] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 413.504348] Lustre: 19519:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 34322886-4cb2-4120-a5d6-83847305eedf@ [ 413.512184] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 413.520310] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 413.520435] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 413.530036] Lustre: Skipped 2 previous similar messages [ 413.548452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 413.548460] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 419.962858] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 17:10:39 (1778361039) [ 423.435557] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 424.351913] Lustre: Failing over lustre-MDT0000 [ 424.452625] Lustre: server umount lustre-MDT0000 complete [ 425.112491] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 425.124526] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 427.487966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 427.496984] Lustre: Skipped 5 previous similar messages [ 440.024864] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 440.027600] LDISKFS-fs (dm-0): recovery complete [ 440.032366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.081172] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 440.186085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.461847] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 441.794856] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 445.422436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 445.432850] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 445.461160] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 445.461251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 447.101793] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 447.866575] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 452.224975] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 17:11:12 (1778361072) [ 455.689607] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 456.494440] Lustre: Failing over lustre-MDT0000 [ 456.664863] Lustre: server umount lustre-MDT0000 complete [ 460.768584] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.768804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 460.776906] Lustre: Skipped 2 previous similar messages [ 472.324813] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 472.328416] LDISKFS-fs (dm-0): recovery complete [ 472.335504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.399028] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 474.074721] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 476.301355] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 477.667170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 477.671483] Lustre: Skipped 3 previous similar messages [ 477.708981] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 477.737575] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:131 to 0x2c0000401:161) [ 477.737632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:161) [ 479.391604] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 480.166547] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 484.587644] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 17:11:44 (1778361104) [ 488.054770] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 489.008720] Lustre: Failing over lustre-MDT0000 [ 489.189562] Lustre: server umount lustre-MDT0000 complete [ 493.023834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.024859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 493.032302] Lustre: Skipped 2 previous similar messages [ 504.840146] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 504.842860] LDISKFS-fs (dm-0): recovery complete [ 504.848957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 504.915651] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 506.751965] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 507.021684] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 510.479362] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 510.514214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:163 to 0x2c0000401:193) [ 510.514236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:193) [ 512.304083] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 513.160708] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 517.200733] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 17:12:17 (1778361137) [ 520.336030] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 520.884847] Lustre: Failing over lustre-MDT0000 [ 520.999939] Lustre: server umount lustre-MDT0000 complete [ 525.792762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 536.319248] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 536.320567] LDISKFS-fs (dm-0): recovery complete [ 536.323799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.366312] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.849898] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 541.668563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 541.672820] Lustre: Skipped 8 previous similar messages [ 541.735156] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 541.735159] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 543.386572] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 544.215200] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 548.745538] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 17:12:48 (1778361168) [ 552.201468] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 553.156907] Lustre: Failing over lustre-MDT0000 [ 553.358726] Lustre: server umount lustre-MDT0000 complete [ 557.024579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.025128] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.032880] Lustre: Skipped 8 previous similar messages [ 557.043064] LustreError: 6484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 61 previous similar messages [ 569.168560] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 569.171628] LDISKFS-fs (dm-0): recovery complete [ 569.176937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.232289] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.349528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 569.353111] Lustre: Skipped 3 previous similar messages [ 571.258921] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 573.583251] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 573.588780] Lustre: Skipped 1 previous similar message [ 574.460162] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 574.468401] Lustre: Skipped 1 previous similar message [ 574.500628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 574.500650] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 576.185384] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 576.776975] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 581.420322] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 17:13:21 (1778361201) [ 581.908657] Lustre: *** cfs_fail_loc=13b, val=315*** [ 581.911709] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 581.914798] LustreError: 6486:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f077a4be680 x1864746303347584/t38654705666(0) o35->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:237/0 lens 392/456 e 0 to 0 dl 1778361212 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 586.626661] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 587.579790] Lustre: Failing over lustre-MDT0000 [ 587.861306] Lustre: server umount lustre-MDT0000 complete [ 589.792665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 589.799466] LustreError: Skipped 1 previous similar message [ 603.637425] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 603.640040] LDISKFS-fs (dm-0): recovery complete [ 603.647243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.845625] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 605.646512] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 609.277040] Lustre: 6486:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f06496c9c00 x1864746303347584/t38654705666(0) o35->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:265/0 lens 392/456 e 0 to 0 dl 1778361240 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 609.292962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 609.293073] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 610.942888] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 611.805542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 616.228395] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 17:13:56 (1778361236) [ 619.749311] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 620.669487] Lustre: Failing over lustre-MDT0000 [ 620.820343] Lustre: server umount lustre-MDT0000 complete [ 636.385554] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 636.389646] LDISKFS-fs (dm-0): recovery complete [ 636.395289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.442591] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.445509] LustreError: Skipped 1 previous similar message [ 636.536747] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 638.016933] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 640.141211] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 640.146887] Lustre: Skipped 1 previous similar message [ 642.044662] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 642.051133] Lustre: Skipped 1 previous similar message [ 642.078985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 642.079118] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 643.686912] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 644.415814] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 648.992155] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 17:14:28 (1778361268) [ 652.537869] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 652.995958] Lustre: *** cfs_fail_loc=114, val=0*** [ 654.262662] Lustre: Failing over lustre-MDT0000 [ 654.437576] Lustre: server umount lustre-MDT0000 complete [ 657.376263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 657.381690] LustreError: Skipped 1 previous similar message [ 670.168094] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 670.170868] LDISKFS-fs (dm-0): recovery complete [ 670.176392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 670.315828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 671.895237] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 675.817155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 675.824188] Lustre: Skipped 14 previous similar messages [ 675.859246] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 675.859397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 677.601963] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 678.468586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 682.634145] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 17:15:02 (1778361302) [ 686.338385] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 686.817078] Lustre: *** cfs_fail_loc=128, val=0*** [ 688.169948] Lustre: Failing over lustre-MDT0000 [ 688.344974] Lustre: server umount lustre-MDT0000 complete [ 691.168323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 691.176236] Lustre: Skipped 14 previous similar messages [ 704.287646] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 704.290989] LDISKFS-fs (dm-0): recovery complete [ 704.296678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 704.498330] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 706.032507] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 709.651842] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 709.651925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 711.212709] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 711.929291] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 715.875607] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 17:15:35 (1778361335) [ 719.281172] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 720.242585] Lustre: Failing over lustre-MDT0000 [ 720.414974] Lustre: server umount lustre-MDT0000 complete [ 736.025120] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 736.027628] LDISKFS-fs (dm-0): recovery complete [ 736.031995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.183116] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 737.619583] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 741.454907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 741.454934] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 743.072227] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 743.811230] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 748.213071] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 17:16:08 (1778361368) [ 751.696372] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 752.703874] Lustre: Failing over lustre-MDT0000 [ 752.782210] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 753.029204] Lustre: server umount lustre-MDT0000 complete [ 768.789681] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 768.792612] LDISKFS-fs (dm-0): recovery complete [ 768.797836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.847026] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.851137] LustreError: Skipped 3 previous similar messages [ 768.968708] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 770.544950] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 773.261734] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 773.267644] Lustre: Skipped 3 previous similar messages [ 774.162794] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 774.168573] Lustre: Skipped 3 previous similar messages [ 774.194826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 774.194851] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 775.859744] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 776.574809] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 781.262299] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 17:16:41 (1778361401) [ 784.658739] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 787.911652] Lustre: Failing over lustre-MDT0000 [ 788.094248] Lustre: server umount lustre-MDT0000 complete [ 789.472720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 789.477851] LustreError: Skipped 3 previous similar messages [ 803.786379] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 803.789286] LDISKFS-fs (dm-0): recovery complete [ 803.794726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.966647] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 805.656646] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 810.607469] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 810.607503] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 812.344619] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 813.184845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 825.106411] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 17:17:24 (1778361444) [ 828.801285] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 829.724460] Lustre: Failing over lustre-MDT0000 [ 829.920055] LustreError: 10951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 829.925398] LustreError: 10951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 119 previous similar messages [ 829.934708] Lustre: server umount lustre-MDT0000 complete [ 845.529075] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 845.531589] LDISKFS-fs (dm-0): recovery complete [ 845.536352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 845.741708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 845.745240] Lustre: Skipped 7 previous similar messages [ 845.768691] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 847.546311] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 850.950282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 850.950419] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 852.684097] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 853.460986] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 860.149256] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 17:17:59 (1778361479) [ 863.761367] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 864.722154] Lustre: Failing over lustre-MDT0000 [ 864.952956] Lustre: server umount lustre-MDT0000 complete [ 880.442954] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 880.445874] LDISKFS-fs (dm-0): recovery complete [ 880.450993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 882.348566] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 885.779759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 885.779795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 887.310355] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 888.152966] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 892.954300] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 17:18:32 (1778361512) [ 896.653714] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 897.592227] Lustre: Failing over lustre-MDT0000 [ 897.761578] Lustre: server umount lustre-MDT0000 complete [ 913.566183] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 913.568949] LDISKFS-fs (dm-0): recovery complete [ 913.574384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.760746] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 913.764185] Lustre: Skipped 1 previous similar message [ 915.588662] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 919.060102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 919.060167] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 920.776303] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 921.593173] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 926.057835] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 17:19:05 (1778361545) [ 929.580245] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 930.520856] Lustre: Failing over lustre-MDT0000 [ 930.699822] Lustre: server umount lustre-MDT0000 complete [ 946.221457] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 946.224365] LDISKFS-fs (dm-0): recovery complete [ 946.229811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 948.047555] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 951.779234] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 951.783626] Lustre: Skipped 31 previous similar messages [ 951.846984] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 951.847083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 953.380917] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 954.110397] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 958.729591] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 17:19:38 (1778361578) [ 962.240304] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 963.176249] Lustre: Failing over lustre-MDT0000 [ 963.426401] Lustre: server umount lustre-MDT0000 complete [ 967.136500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.144939] Lustre: Skipped 31 previous similar messages [ 979.024950] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 979.027717] LDISKFS-fs (dm-0): recovery complete [ 979.032880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.804206] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 984.591836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 984.591842] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 985.997857] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 986.693231] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 990.679744] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 17:20:10 (1778361610) [ 994.057096] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 994.960611] Lustre: Failing over lustre-MDT0000 [ 995.246321] Lustre: server umount lustre-MDT0000 complete [ 1010.966908] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1010.969573] LDISKFS-fs (dm-0): recovery complete [ 1010.974766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.845437] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1016.340904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1016.340939] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1017.994649] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1018.803682] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1023.234646] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 17:20:43 (1778361643) [ 1026.909755] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1027.886291] Lustre: Failing over lustre-MDT0000 [ 1028.178818] Lustre: server umount lustre-MDT0000 complete [ 1044.006060] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1044.008951] LDISKFS-fs (dm-0): recovery complete [ 1044.014475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1044.069865] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1044.073180] LustreError: Skipped 7 previous similar messages [ 1044.204551] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1044.207052] Lustre: Skipped 3 previous similar messages [ 1044.624802] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1044.630920] Lustre: Skipped 7 previous similar messages [ 1045.996609] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1049.617716] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1049.623431] Lustre: Skipped 7 previous similar messages [ 1049.650803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1049.650916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1051.296162] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1052.070598] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1056.534676] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 17:21:16 (1778361676) [ 1059.922206] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1060.873452] Lustre: Failing over lustre-MDT0000 [ 1061.139880] Lustre: server umount lustre-MDT0000 complete [ 1064.929566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1064.934462] LustreError: Skipped 6 previous similar messages [ 1077.181418] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1077.184480] LDISKFS-fs (dm-0): recovery complete [ 1077.189820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.228906] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1082.912437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1082.912477] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1084.602837] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1085.409953] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1089.898886] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 17:21:49 (1778361709) [ 1093.659873] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1094.592153] Lustre: Failing over lustre-MDT0000 [ 1094.803604] Lustre: server umount lustre-MDT0000 complete [ 1110.640901] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1110.643835] LDISKFS-fs (dm-0): recovery complete [ 1110.649612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1112.729339] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1116.171833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 1116.171897] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 1117.694207] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1118.531053] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1123.294393] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 17:22:23 (1778361743) [ 1127.054660] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1128.018251] Lustre: Failing over lustre-MDT0000 [ 1128.254692] Lustre: server umount lustre-MDT0000 complete [ 1144.105190] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1144.107915] LDISKFS-fs (dm-0): recovery complete [ 1144.112716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.916222] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1149.454374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:867 to 0x280000401:897) [ 1149.454413] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:897) [ 1151.133234] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1151.928266] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1156.311696] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 17:22:56 (1778361776) [ 1159.966366] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1160.871369] Lustre: Failing over lustre-MDT0000 [ 1161.147762] Lustre: server umount lustre-MDT0000 complete [ 1176.743274] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1176.746208] LDISKFS-fs (dm-0): recovery complete [ 1176.751548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.463748] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1182.227485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1182.227527] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1183.936669] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1184.770654] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1189.213826] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 17:23:29 (1778361809) [ 1192.816204] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1193.747904] Lustre: Failing over lustre-MDT0000 [ 1194.056705] Lustre: server umount lustre-MDT0000 complete [ 1209.815944] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1209.818877] LDISKFS-fs (dm-0): recovery complete [ 1209.824332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.591476] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1215.500407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 1215.500434] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 1217.159952] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1217.912441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1222.429482] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 17:24:02 (1778361842) [ 1225.969358] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1226.878060] Lustre: Failing over lustre-MDT0000 [ 1227.171198] Lustre: server umount lustre-MDT0000 complete [ 1242.860410] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1242.863228] LDISKFS-fs (dm-0): recovery complete [ 1242.869036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.766461] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1248.271395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:963 to 0x280000401:993) [ 1248.271412] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:993) [ 1249.934810] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1250.764826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1255.430350] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 17:24:35 (1778361875) [ 1259.039600] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1260.036895] Lustre: Failing over lustre-MDT0000 [ 1260.304141] Lustre: server umount lustre-MDT0000 complete [ 1275.971499] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1275.974403] LDISKFS-fs (dm-0): recovery complete [ 1275.979760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.741766] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1281.558445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1281.558721] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1283.196183] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1283.946696] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1288.688146] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 17:25:08 (1778361908) [ 1292.240132] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1293.228380] Lustre: Failing over lustre-MDT0000 [ 1293.511507] Lustre: server umount lustre-MDT0000 complete [ 1309.262432] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1309.265277] LDISKFS-fs (dm-0): recovery complete [ 1309.270754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1309.410032] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1309.412741] Lustre: Skipped 7 previous similar messages [ 1311.143328] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1314.832974] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1314.833047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1316.514461] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1317.340226] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1321.854714] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 17:25:41 (1778361941) [ 1325.308573] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1326.253465] Lustre: Failing over lustre-MDT0000 [ 1326.537592] Lustre: server umount lustre-MDT0000 complete [ 1342.245543] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1342.248238] LDISKFS-fs (dm-0): recovery complete [ 1342.253168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.073754] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1347.599881] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 1347.599972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 1349.251826] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1350.038910] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1354.039140] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 17:26:13 (1778361973) [ 1356.113309] Lustre: 71744:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 at adminstrative request [ 1358.665377] Lustre: Failing over lustre-MDT0000 [ 1358.934472] Lustre: server umount lustre-MDT0000 complete [ 1362.062108] LustreError: 6483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.070670] LustreError: 6483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 240 previous similar messages [ 1372.781871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1372.921351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1372.923887] Lustre: Skipped 15 previous similar messages [ 1374.556824] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1378.305176] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 1378.305268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1379.914692] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1380.681323] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1383.528427] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1387.950135] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1391.077611] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 17:26:50 (1778362010) [ 1391.582638] Lustre: 73641:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 at adminstrative request [ 1396.030519] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 17:26:55 (1778362015) [ 1399.572822] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1400.519353] Lustre: Failing over lustre-MDT0000 [ 1400.810282] Lustre: server umount lustre-MDT0000 complete [ 1416.416597] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1416.419137] LDISKFS-fs (dm-0): recovery complete [ 1416.423454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1418.160955] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1421.852293] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1124 to 0x2c0000401:1153) [ 1421.852350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1123 to 0x280000401:1153) [ 1423.358266] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1424.181315] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1428.745686] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 17:27:28 (1778362048) [ 1432.151478] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1433.012536] Lustre: Failing over lustre-MDT0000 [ 1433.204596] Lustre: server umount lustre-MDT0000 complete [ 1448.484928] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1448.487826] LDISKFS-fs (dm-0): recovery complete [ 1448.493190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.308795] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1454.086079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1454.086111] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1455.831924] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1456.660234] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1461.462555] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 17:28:01 (1778362081) [ 1465.127413] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1466.071188] Lustre: Failing over lustre-MDT0000 [ 1466.366985] Lustre: server umount lustre-MDT0000 complete [ 1481.583350] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1481.587221] LDISKFS-fs (dm-0): recovery complete [ 1481.593832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.526824] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1486.819171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1486.824435] Lustre: Skipped 63 previous similar messages [ 1486.862443] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1486.862464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1488.534669] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1489.332718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1493.828369] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 17:28:33 (1778362113) [ 1497.321258] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1498.224600] Lustre: Failing over lustre-MDT0000 [ 1498.417341] Lustre: server umount lustre-MDT0000 complete [ 1502.176164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1502.184548] Lustre: Skipped 63 previous similar messages [ 1514.120507] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1514.123313] LDISKFS-fs (dm-0): recovery complete [ 1514.128573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1515.937918] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1519.613208] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1519.613225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1521.405044] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1522.239625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1526.857754] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 17:29:06 (1778362146) [ 1530.474961] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1531.407694] Lustre: Failing over lustre-MDT0000 [ 1531.700976] Lustre: server umount lustre-MDT0000 complete [ 1547.450074] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1547.453209] LDISKFS-fs (dm-0): recovery complete [ 1547.458820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.347292] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1552.911862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1281) [ 1552.911890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1251 to 0x2c0000401:1281) [ 1554.581656] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1555.376831] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1559.810971] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 17:29:39 (1778362179) [ 1563.333175] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1564.317215] Lustre: Failing over lustre-MDT0000 [ 1564.511131] Lustre: server umount lustre-MDT0000 complete [ 1580.207413] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1580.210338] LDISKFS-fs (dm-0): recovery complete [ 1580.215555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1580.273411] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1580.279789] LustreError: Skipped 15 previous similar messages [ 1582.094237] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1582.220880] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1582.223523] Lustre: Skipped 15 previous similar messages [ 1585.670734] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 1585.676565] Lustre: Skipped 15 previous similar messages [ 1585.702757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1585.702779] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1587.371911] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1588.211578] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1592.151508] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 17:30:12 (1778362212) [ 1595.464434] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1596.465157] Lustre: Failing over lustre-MDT0000 [ 1596.743684] Lustre: server umount lustre-MDT0000 complete [ 1600.992771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1600.997875] LustreError: Skipped 15 previous similar messages [ 1612.419184] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1612.422123] LDISKFS-fs (dm-0): recovery complete [ 1612.427377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1614.140288] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1617.920321] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1617.920351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1619.405665] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1620.051168] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1624.017545] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 17:30:43 (1778362243) [ 1627.165414] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1628.071502] Lustre: Failing over lustre-MDT0000 [ 1628.127747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1628.130056] Lustre: Skipped 3 previous similar messages [ 1628.345464] Lustre: server umount lustre-MDT0000 complete [ 1642.177214] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1642.178407] LDISKFS-fs (dm-0): recovery complete [ 1642.181084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1643.447883] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1647.606640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 1647.606646] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 1648.571690] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1649.033980] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1652.281788] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 17:31:12 (1778362272) [ 1655.120859] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1656.037885] Lustre: Failing over lustre-MDT0000 [ 1656.201042] Lustre: server umount lustre-MDT0000 complete [ 1671.301546] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1671.304243] LDISKFS-fs (dm-0): recovery complete [ 1671.308791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.945154] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1676.809719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 1676.809774] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 1678.473350] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1679.239449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1683.717227] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 17:31:43 (1778362303) [ 1687.240135] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1688.038742] Lustre: Failing over lustre-MDT0000 [ 1688.290575] Lustre: server umount lustre-MDT0000 complete [ 1703.504068] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1703.506535] LDISKFS-fs (dm-0): recovery complete [ 1703.510772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1704.846243] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1709.045521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 1709.045526] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 1710.190890] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1710.742842] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1713.991554] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 17:32:14 (1778362334) [ 1716.341967] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1716.913838] Lustre: Failing over lustre-MDT0000 [ 1717.124680] Lustre: server umount lustre-MDT0000 complete [ 1730.761452] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1730.762896] LDISKFS-fs (dm-0): recovery complete [ 1730.766189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1732.033984] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1736.181546] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 1736.181555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 1737.482868] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1738.158420] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1742.329019] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 17:32:42 (1778362362) [ 1742.758282] Lustre: 93998:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 at adminstrative request [ 1747.917923] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 17:32:47 (1778362367) [ 1750.978447] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1751.893114] Lustre: Failing over lustre-MDT0000 [ 1752.165462] Lustre: server umount lustre-MDT0000 complete [ 1757.981367] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1757.984348] LDISKFS-fs (dm-0): recovery complete [ 1757.989738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1758.128625] Lustre: lustre-MDT0000: Aborting client recovery [ 1758.130230] LustreError: 95279:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1758.132746] Lustre: 95318:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1758.136415] Lustre: 95318:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@ [ 1758.141765] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1758.145365] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 1758.149723] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 1758.173831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1505) [ 1758.173837] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1505) [ 1759.793113] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1763.297125] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1773.491888] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 17:33:13 (1778362393) [ 1777.013907] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1777.936034] Lustre: Failing over lustre-MDT0000 [ 1778.161393] Lustre: server umount lustre-MDT0000 complete [ 1783.944187] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1783.946981] LDISKFS-fs (dm-0): recovery complete [ 1783.952241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.139406] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1784.187734] Lustre: lustre-MDT0000: Aborting client recovery [ 1784.189464] LustreError: 97133:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1784.192429] Lustre: 97169:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1784.195343] Lustre: 97169:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1784.197893] Lustre: 97169:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1784.201422] Lustre: 97169:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1784.204966] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1784.208440] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 1784.216032] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 1784.243717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1537) [ 1784.243722] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1475 to 0x2c0000401:1537) [ 1785.903837] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1789.409782] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1796.182539] Lustre: *** cfs_fail_loc=1311, val=0*** [ 1796.185240] Lustre: Skipped 1 previous similar message [ 1799.602145] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 17:33:39 (1778362419) [ 1803.191595] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1804.036345] Lustre: Failing over lustre-MDT0000 [ 1804.157487] Lustre: server umount lustre-MDT0000 complete [ 1810.013632] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1810.016461] LDISKFS-fs (dm-0): recovery complete [ 1810.021864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.194502] Lustre: lustre-MDT0000: Aborting client recovery [ 1810.196303] LustreError: 98987:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1810.199071] Lustre: 99024:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1810.201964] Lustre: 99024:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1810.205341] Lustre: 99024:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1810.211117] Lustre: 99024:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1810.215120] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1810.218790] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 1810.226569] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 1810.255856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 1810.255895] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 1811.827342] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1815.524947] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1825.197630] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 17:34:04 (1778362444) [ 1825.697797] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 1825.701973] LustreError: 8392:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f0781f6e300 x1864746304279552/t201863462916(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:726/0 lens 512/456 e 0 to 0 dl 1778362456 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1828.548670] Lustre: Failing over lustre-MDT0000 [ 1828.672445] Lustre: server umount lustre-MDT0000 complete [ 1832.873621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.096849] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1833.096938] Lustre: lustre-MDT0000: Aborting client recovery [ 1833.099124] Lustre: Skipped 22 previous similar messages [ 1833.104534] LustreError: 100301:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1833.108515] Lustre: 100337:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1833.113184] Lustre: 100337:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1833.118164] Lustre: 100337:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1833.124912] Lustre: 100337:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 1833.129679] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1833.134306] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 1833.144231] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 1833.170189] Lustre: 9449:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f0750729180 x1864746304279552/t201863462916(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:733/0 lens 512/2888 e 0 to 0 dl 1778362463 ref 1 fl Interpret:/202/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 1833.180806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 1833.180822] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1601) [ 1834.779159] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1838.048456] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1847.977455] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 1848.845710] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 17:34:28 (1778362468) [ 1852.441259] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1853.719062] Lustre: Failing over lustre-MDT0000 [ 1853.996367] Lustre: server umount lustre-MDT0000 complete [ 1859.870126] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1859.873135] LDISKFS-fs (dm-0): recovery complete [ 1859.878820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.070919] Lustre: lustre-MDT0000: Aborting client recovery [ 1860.073360] LustreError: 102251:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1860.077222] Lustre: 102287:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1860.080493] Lustre: 102287:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 1860.083906] Lustre: 102287:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 1860.088790] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 1860.092176] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 1860.099874] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 1860.130234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 1860.130281] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1633) [ 1861.725189] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1865.186243] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 1875.859667] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 17:34:55 (1778362495) [ 1888.707463] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1889.630860] Lustre: Failing over lustre-MDT0000 [ 1889.847202] Lustre: server umount lustre-MDT0000 complete [ 1905.432141] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1905.434793] LDISKFS-fs (dm-0): recovery complete [ 1905.439380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.373678] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1910.823607] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 1910.823652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 1912.670496] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1913.498762] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1924.270550] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 17:35:44 (1778362544) [ 1932.545220] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1935.396347] Lustre: Failing over lustre-MDT0000 [ 1935.501941] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 1935.504096] Lustre: Skipped 1 previous similar message [ 1935.715346] Lustre: server umount lustre-MDT0000 complete [ 1949.914366] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1949.915642] LDISKFS-fs (dm-0): recovery complete [ 1949.918970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.251607] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1955.895552] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 1955.898976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 1956.925374] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1957.400624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1966.073556] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 17:36:25 (1778362585) [ 1967.008353] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1967.479829] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1967.485089] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1970.661245] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 17:36:30 (1778362590) [ 1980.057817] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1984.061345] Lustre: Failing over lustre-OST0000 [ 1984.153335] Lustre: server umount lustre-OST0000 complete [ 1986.017628] LustreError: 42095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1986.032606] LustreError: 42095:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 232 previous similar messages [ 1999.544698] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 1999.548064] LDISKFS-fs (dm-2): recovery complete [ 1999.553159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1999.626926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.631927] Lustre: Skipped 18 previous similar messages [ 2002.148327] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2050.439834] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 17:37:50 (1778362670) [ 2054.233735] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2055.607278] Lustre: Failing over lustre-MDT0000 [ 2055.907505] Lustre: server umount lustre-MDT0000 complete [ 2071.519179] Lustre: 3632:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778362675/real 1778362675] req@ffff9f064736c380 x1864746312414720/t0(0) o400->MGC192.168.204.132@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1778362691 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2071.894525] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2071.897337] LDISKFS-fs (dm-0): recovery complete [ 2071.903340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.760780] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f07659c9880 x1864746312422656/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2083.408978] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2087.396363] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2087.400533] Lustre: Skipped 66 previous similar messages [ 2087.442914] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2087.443317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 2089.012346] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2089.837730] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2103.775137] Lustre: 111370:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778362707/real 1778362707] req@ffff9f064b383800 x1864746312472192/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1778362723 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 2103.784378] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.789115] Lustre: Skipped 64 previous similar messages [ 2103.790847] LustreError: 111370:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2103.792191] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2104.619428] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 17:38:44 (1778362724) [ 2104.799728] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 2107.167560] LustreError: 9449:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2112.479179] LustreError: 9449:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2112.484720] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnecting [ 2113.375853] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2118.623150] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2118.628933] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnecting [ 2119.524825] LustreError: 26179:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2124.767114] LustreError: 26179:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2124.772519] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnecting [ 2125.644551] LustreError: 9449:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2130.911169] LustreError: 9449:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2131.782374] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2137.055132] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2137.060923] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnecting [ 2137.066537] Lustre: Skipped 1 previous similar message [ 2143.814306] LustreError: 6484:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2143.817740] LustreError: 6484:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 2148.831170] LustreError: 6484:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2148.837528] LustreError: 6484:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 1 previous similar message [ 2154.975231] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnecting [ 2154.982828] Lustre: Skipped 2 previous similar messages [ 2162.001866] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_race id 701 sleeping [ 2162.008540] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 2167.263145] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2167.268219] LustreError: 6485:0:(ldlm_lib.c:1165:target_handle_connect()) Skipped 2 previous similar messages [ 2171.354868] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 17:39:51 (1778362791) [ 2172.279821] LustreError: 9449:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2181.773564] Lustre: lustre-MDT0000: Export ffff9f075dffc000 already connecting from 192.168.204.32@tcp [ 2186.893431] Lustre: lustre-MDT0000: Export ffff9f075dffc000 already connecting from 192.168.204.32@tcp [ 2192.013284] Lustre: lustre-MDT0000: Export ffff9f075dffc000 already connecting from 192.168.204.32@tcp [ 2197.133318] Lustre: lustre-MDT0000: Export ffff9f075dffc000 already connecting from 192.168.204.32@tcp [ 2197.138740] Lustre: Skipped 1 previous similar message [ 2202.253292] Lustre: lustre-MDT0000: Export ffff9f075dffc000 already connecting from 192.168.204.32@tcp [ 2212.327147] LustreError: 9449:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2212.333282] Lustre: 9449:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9f0646447800 x1864746306914432/t0(0) o38->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:0/0 lens 520/416 e 0 to 0 dl 1778362812 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2212.493337] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnecting [ 2212.499442] Lustre: Skipped 3 previous similar messages [ 2212.502410] LustreError: 6483:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2237.069507] Lustre: lustre-MDT0000: Export ffff9f075dffc000 already connecting from 192.168.204.32@tcp [ 2237.075187] Lustre: Skipped 1 previous similar message [ 2252.551095] LustreError: 6483:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2252.555620] Lustre: 6483:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9f077a598a80 x1864746306918912/t0(0) o38->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:0/0 lens 520/416 e 0 to 0 dl 1778362852 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2256.802541] LustreError: 8392:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2281.612754] Lustre: lustre-MDT0000: Export ffff9f075dffc000 already connecting from 192.168.204.32@tcp [ 2281.616219] Lustre: Skipped 3 previous similar messages [ 2296.847120] LustreError: 8392:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2296.849738] Lustre: 8392:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9f077a633100 x1864746306922496/t0(0) o38->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:0/0 lens 520/416 e 0 to 0 dl 1778362896 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2296.972717] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnecting [ 2296.975387] Lustre: Skipped 1 previous similar message [ 2296.976647] LustreError: 26179:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2321.548921] Lustre: lustre-MDT0000: Export ffff9f075dffc000 already connecting from 192.168.204.32@tcp [ 2321.553220] Lustre: Skipped 2 previous similar messages [ 2337.023124] LustreError: 26179:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2337.025479] Lustre: 26179:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9f0765886680 x1864746306925440/t0(0) o38->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:0/0 lens 520/416 e 0 to 0 dl 1778362936 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2340.588223] LustreError: 6483:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2380.631136] LustreError: 6483:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2380.634056] Lustre: 6483:0:(service.c:2561:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9f0781e60a80 x1864746306929024/t0(0) o38->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:0/0 lens 520/416 e 0 to 0 dl 1778362980 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2381.453091] LustreError: 9449:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2382.703061] LustreError: 9449:0:(ldlm_lib.c:1419:target_handle_connect()) cfs_fail_timeout interrupted [ 2384.172867] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 17:43:24 (1778363004) [ 2386.689544] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2387.922677] Lustre: Failing over lustre-MDT0000 [ 2388.016273] Lustre: server umount lustre-MDT0000 complete [ 2389.472428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2389.474973] LustreError: Skipped 11 previous similar messages [ 2392.241636] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2392.243777] LDISKFS-fs (dm-0): recovery complete [ 2392.246895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.287358] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2392.290738] LustreError: Skipped 13 previous similar messages [ 2392.350098] Lustre: *** cfs_fail_loc=712, val=0*** [ 2392.352794] LustreError: 42096:0:(service.c:1390:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9f0765837480 x1864746312620928/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 2392.409424] Lustre: lustre-MDT0000: Aborting client recovery [ 2392.410921] LustreError: 116102:0:(ldlm_lib.c:2984:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2392.413291] Lustre: 116139:0:(ldlm_lib.c:2387:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2392.415799] Lustre: 116139:0:(ldlm_lib.c:2387:target_recovery_overseer()) Skipped 2 previous similar messages [ 2392.418125] Lustre: 116139:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2392.421426] Lustre: 116139:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2392.423739] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2392.426464] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 2392.432242] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 2392.451891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 2392.454461] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 2393.838397] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2397.665381] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2404.836331] Lustre: Failing over lustre-MDT0000 [ 2405.003365] Lustre: server umount lustre-MDT0000 complete [ 2417.809787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.296666] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2422.412794] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2422.415441] Lustre: Skipped 10 previous similar messages [ 2423.276238] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2423.279422] Lustre: Skipped 10 previous similar messages [ 2423.296272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 2423.296273] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 2424.495961] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2425.004409] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2428.315265] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 17:44:08 (1778363048) [ 2428.365930] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnecting [ 2428.368470] Lustre: Skipped 2 previous similar messages [ 2431.478296] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 17:44:11 (1778363051) [ 2431.811568] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 2431.814779] LustreError: 6493:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f0765884a80 x1864746306984960/t0(0) o700->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:577/0 lens 264/248 e 0 to 0 dl 1778363062 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 2449.091948] Lustre: Failing over lustre-MDT0000 [ 2449.257600] Lustre: server umount lustre-MDT0000 complete [ 2461.951492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.117621] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2462.120548] Lustre: Skipped 11 previous similar messages [ 2463.361802] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2467.318073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 2467.318124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 2468.367750] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2468.907420] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2472.595701] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 17:44:52 (1778363092) [ 2473.370856] Lustre: Failing over lustre-OST0000 [ 2473.427995] Lustre: server umount lustre-OST0000 complete [ 2486.051298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2487.932844] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2490.946716] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2491.472790] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2556.903954] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 17:46:16 (1778363176) [ 2559.393085] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2560.116156] Lustre: Failing over lustre-MDT0000 [ 2560.355110] Lustre: server umount lustre-MDT0000 complete [ 2574.450331] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2574.452848] LDISKFS-fs (dm-0): recovery complete [ 2574.456908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2575.927227] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2580.023102] Lustre: *** cfs_fail_loc=216, val=0*** [ 2580.023119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 2580.024677] LustreError: 121924:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 2581.087551] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 2642.545310] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 17:47:42 (1778363262) [ 2650.543532] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 17:47:50 (1778363270) [ 2651.575575] Lustre: Failing over lustre-MDT0000 [ 2651.616587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.618558] Lustre: Skipped 3 previous similar messages [ 2651.753316] Lustre: server umount lustre-MDT0000 complete [ 2652.813985] LustreError: 9449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2652.818935] LustreError: 9449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 85 previous similar messages [ 2664.514376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.630139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2664.633882] Lustre: Skipped 6 previous similar messages [ 2665.877551] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2670.056927] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 2670.059179] LustreError: 123495:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f0765ba4380 x1864746307104128/t0(0) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:103/0 lens 328/344 e 0 to 0 dl 1778363343 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 2727.574289] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnected, waiting for 2 clients in recovery for 1:04 [ 2727.580041] Lustre: 123495:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 2727.589369] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2727.592035] Lustre: Skipped 27 previous similar messages [ 2727.605259] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3052 to 0x2c0000401:3073) [ 2727.605260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3105) [ 2728.968287] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2729.470824] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2733.211578] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 17:49:13 (1778363353) [ 2734.637447] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2737.396210] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2738.023877] Lustre: Failing over lustre-MDT0000 [ 2738.127716] Lustre: server umount lustre-MDT0000 complete [ 2738.143368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2738.147375] Lustre: Skipped 24 previous similar messages [ 2752.032341] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2752.034137] LDISKFS-fs (dm-0): recovery complete [ 2752.037741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.424862] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2757.622975] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 2757.622975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3137) [ 2758.830452] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2759.412498] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2762.923949] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 17:49:42 (1778363382) [ 2763.305120] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2767.087580] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2767.744176] Lustre: Failing over lustre-MDT0000 [ 2767.845676] Lustre: server umount lustre-MDT0000 complete [ 2781.994536] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2781.996497] LDISKFS-fs (dm-0): recovery complete [ 2782.000546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.540431] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2787.322850] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3107 to 0x2c0000401:3137) [ 2787.322910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3169) [ 2788.467745] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2789.025903] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2792.428581] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 17:50:12 (1778363412) [ 2792.820442] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2796.525742] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2797.142091] Lustre: Failing over lustre-MDT0000 [ 2797.230740] Lustre: server umount lustre-MDT0000 complete [ 2811.366977] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2811.368419] LDISKFS-fs (dm-0): recovery complete [ 2811.371439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2812.806918] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2816.503754] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3139 to 0x2c0000401:3169) [ 2816.503796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3201) [ 2820.934114] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 17:50:40 (1778363440) [ 2822.283046] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2822.284329] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2822.285777] LustreError: 38284:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f0765af3800 x1864746307155584/t257698037777(0) o35->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:213/0 lens 392/456 e 0 to 0 dl 1778363453 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2823.168112] Lustre: Failing over lustre-MDT0000 [ 2823.328943] Lustre: server umount lustre-MDT0000 complete [ 2836.164341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.698153] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2841.581888] Lustre: 6487:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f0779eef480 x1864746307155584/t257698037777(0) o35->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:232/0 lens 392/456 e 0 to 0 dl 1778363472 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2841.589140] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3171 to 0x2c0000401:3201) [ 2841.589196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3233) [ 2842.825067] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2843.432441] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2846.874105] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 17:51:06 (1778363466) [ 2847.238803] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2847.240173] LustreError: 6484:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f075dcd1880 x1864746307167872/t261993005072(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:282/0 lens 504/448 e 0 to 0 dl 1778363522 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2850.947361] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2851.567135] Lustre: Failing over lustre-MDT0000 [ 2851.653597] Lustre: server umount lustre-MDT0000 complete [ 2865.648778] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2865.650294] LDISKFS-fs (dm-0): recovery complete [ 2865.655431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.015135] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2871.277947] Lustre: 6484:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f0781ec4e00 x1864746307167872/t261993005072(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:306/0 lens 504/2880 e 0 to 0 dl 1778363546 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2871.282350] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 2871.282368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3265) [ 2872.367492] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2872.927117] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2876.184351] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 17:51:36 (1778363496) [ 2876.520136] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2876.522838] LustreError: 6485:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f076586aa00 x1864746307181568/t266287972368(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:311/0 lens 504/448 e 0 to 0 dl 1778363551 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2877.795859] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2880.067905] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2880.680623] Lustre: Failing over lustre-MDT0000 [ 2880.763631] Lustre: server umount lustre-MDT0000 complete [ 2894.531686] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2894.532928] LDISKFS-fs (dm-0): recovery complete [ 2894.535862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.901090] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2899.950936] Lustre: 6483:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f07659ca680 x1864746307181568/t266287972368(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:334/0 lens 504/2880 e 0 to 0 dl 1778363574 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2899.957892] Lustre: 6483:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 2899.958344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3297) [ 2899.958384] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 2904.313627] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 17:52:04 (1778363524) [ 2904.648915] LustreError: 6483:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f076586ad80 x1864746307194112/t270582939664(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:339/0 lens 504/448 e 0 to 0 dl 1778363579 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2904.655486] LustreError: 6483:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 2905.917835] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 2905.920420] Lustre: Skipped 1 previous similar message [ 2908.565159] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2909.191375] Lustre: Failing over lustre-MDT0000 [ 2909.276548] Lustre: server umount lustre-MDT0000 complete [ 2923.493162] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2923.494614] LDISKFS-fs (dm-0): recovery complete [ 2923.497457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.000049] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2928.625964] Lustre: 6484:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f0765868e00 x1864746307194112/t270582939664(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:363/0 lens 504/2880 e 0 to 0 dl 1778363603 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 2928.636925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3329) [ 2928.636926] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 2932.710812] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 17:52:32 (1778363552) [ 2933.075643] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 2934.358108] Lustre: *** cfs_fail_loc=13b, val=315*** [ 2934.359765] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 2934.361109] Lustre: Skipped 2 previous similar messages [ 2937.761139] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2938.419708] Lustre: Failing over lustre-MDT0000 [ 2938.514083] Lustre: server umount lustre-MDT0000 complete [ 2952.667215] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2952.668691] LDISKFS-fs (dm-0): recovery complete [ 2952.671258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.085678] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2957.805233] Lustre: 6487:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f064aa7b480 x1864746307206016/t274877906960(0) o35->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:348/0 lens 392/456 e 0 to 0 dl 1778363588 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 2957.810992] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3329) [ 2957.811047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3085 to 0x280000401:3361) [ 2962.226477] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 17:53:02 (1778363582) [ 2962.554432] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 2962.556278] LustreError: 9449:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f064aa78380 x1864746307216256/t279172874255(0) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:397/0 lens 664/608 e 0 to 0 dl 1778363637 ref 1 fl Interpret:/600/0 rc 301/0 job:'touch.0' uid:0 gid:0 projid:0 [ 2962.562801] LustreError: 9449:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 3023.505877] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnecting [ 3023.510256] Lustre: Skipped 4 previous similar messages [ 3023.517103] Lustre: 8392:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f075de66300 x1864746307216256/t279172874255(0) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:458/0 lens 664/3488 e 0 to 0 dl 1778363698 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 3025.690517] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 17:54:05 (1778363645) [ 3028.138775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3028.772787] Lustre: Failing over lustre-MDT0000 [ 3028.858610] Lustre: server umount lustre-MDT0000 complete [ 3029.472426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3029.475204] LustreError: Skipped 12 previous similar messages [ 3042.933665] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3042.936061] LDISKFS-fs (dm-0): recovery complete [ 3042.940738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.978891] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3042.982241] LustreError: Skipped 12 previous similar messages [ 3043.980694] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3043.984191] Lustre: Skipped 12 previous similar messages [ 3044.472801] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3048.429441] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3048.433447] Lustre: Skipped 12 previous similar messages [ 3048.449294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 3048.449322] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3299 to 0x2c0000401:3361) [ 3049.613702] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3050.151753] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3063.695748] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 17:54:43 (1778363683) [ 3066.482583] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3067.091647] Lustre: Failing over lustre-MDT0000 [ 3067.282344] Lustre: server umount lustre-MDT0000 complete [ 3081.367139] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3081.368545] LDISKFS-fs (dm-0): recovery complete [ 3081.371552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.525569] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3081.527900] Lustre: Skipped 12 previous similar messages [ 3082.774487] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3086.862992] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3363 to 0x2c0000401:3393) [ 3086.863080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3425) [ 3088.292286] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3088.795927] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3091.047175] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3096.153272] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 17:55:16 (1778363716) [ 3106.322726] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3106.976150] Lustre: Failing over lustre-MDT0000 [ 3107.223480] Lustre: server umount lustre-MDT0000 complete [ 3121.270944] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3121.272789] LDISKFS-fs (dm-0): recovery complete [ 3121.275618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.670996] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3126.853854] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 3126.853890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 3128.005085] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3128.533975] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3141.874938] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 17:56:01 (1778363761) [ 3145.651068] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3146.279185] Lustre: Failing over lustre-MDT0000 [ 3146.404135] Lustre: server umount lustre-MDT0000 complete [ 3160.442602] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3160.444299] LDISKFS-fs (dm-0): recovery complete [ 3160.446961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3162.026320] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3165.697770] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4675 to 0x2c0000401:4705) [ 3165.697770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4737) [ 3166.852022] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3167.377158] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3170.238873] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 3170.752785] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3172.809870] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 17:56:32 (1778363792) [ 3178.683396] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 3239.343867] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3239.345268] Lustre: Skipped 1 previous similar message [ 3239.346485] LustreError: 6484:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f077a599c00 x1864746310005248/t296352743434(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:674/0 lens 66040/440 e 0 to 0 dl 1778363914 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3298.958494] Lustre: 9449:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f0780b6cc50 x1864746310005248/t296352743434(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:733/0 lens 66040/440 e 0 to 0 dl 1778363973 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 3301.937520] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 3302.484443] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 17:58:42 (1778363922) [ 3306.639315] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3307.439442] Lustre: Failing over lustre-MDT0000 [ 3307.558107] Lustre: server umount lustre-MDT0000 complete [ 3309.024202] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3309.032798] LustreError: 6489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 189 previous similar messages [ 3321.706579] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3321.708382] LDISKFS-fs (dm-0): recovery complete [ 3321.711361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3321.825715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.827665] Lustre: Skipped 12 previous similar messages [ 3323.187128] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3327.167792] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 3327.167792] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 3328.634855] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3329.205668] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3333.321538] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 17:59:13 (1778363953) [ 3338.757648] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 3341.952630] Lustre: Failing over lustre-OST0000 [ 3341.988859] Lustre: server umount lustre-OST0000 complete [ 3342.305237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.309530] Lustre: Skipped 51 previous similar messages [ 3356.601494] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 3356.604642] LDISKFS-fs (dm-2): recovery complete [ 3356.609400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3357.937161] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3357.939617] Lustre: Skipped 52 previous similar messages [ 3358.518717] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3370.132357] Lustre: Failing over lustre-OST0000 [ 3370.184581] Lustre: server umount lustre-OST0000 complete [ 3383.083189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3385.299435] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3388.463578] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3389.100974] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3422.873633] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 18:00:42 (1778364042) [ 3423.782328] Lustre: Failing over lustre-MDT0000 [ 3424.088360] Lustre: server umount lustre-MDT0000 complete [ 3436.960249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.376406] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3442.163841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 3442.163841] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 3449.875113] Lustre: Failing over lustre-MDT0000 [ 3450.060963] Lustre: server umount lustre-MDT0000 complete [ 3463.095898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.485386] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3468.282862] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 3468.282964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 3469.545179] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3470.142832] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3473.973762] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 18:01:33 (1778364093) [ 3485.642222] Lustre: Failing over lustre-OST0000 [ 3485.701184] Lustre: server umount lustre-OST0000 complete [ 3498.589235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3500.803953] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3504.042454] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3504.705468] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3508.790381] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 18:02:08 (1778364128) [ 3509.599344] Lustre: Failing over lustre-MDT0000 [ 3509.911552] Lustre: server umount lustre-MDT0000 complete [ 3513.236127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.265284] Lustre: *** cfs_fail_loc=605, val=0*** [ 3513.266485] LustreError: 157295:0:(llog_obd.c:190:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc08d4b00 failed: rc = -95 [ 3513.269259] LustreError: 157295:0:(obd_config.c:843:class_setup()) setup MGS failed (-95) [ 3513.271103] LustreError: 157295:0:(obd_mount.c:250:lustre_start_simple()) MGS setup error -95 [ 3513.272942] LustreError: 157295:0:(tgt_mount.c:114:server_deregister_mount()) MGS not registered [ 3513.274812] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 3513.276465] LustreError: 157295:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-MDT0000 [ 3513.280616] Lustre: server umount lustre-MDT0000 complete [ 3513.281776] LustreError: 157295:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 3516.020181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.445200] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3520.095707] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 18:02:20 (1778364140) [ 3521.528089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 3521.528109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 3523.939595] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3524.936145] Lustre: Failing over lustre-MDT0000 [ 3525.226203] Lustre: server umount lustre-MDT0000 complete [ 3539.403035] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3539.404508] LDISKFS-fs (dm-0): recovery complete [ 3539.407610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3539.598175] Lustre: *** cfs_fail_loc=707, val=0*** [ 3540.818443] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3600.027864] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 3600.047567] Lustre: 159458:0:(ldlm_lib.c:2068:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3600.051163] Lustre: 159458:0:(ldlm_lib.c:2068:extend_recovery_timer()) Skipped 1 previous similar message [ 3600.140320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 3600.140321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 3601.586521] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3602.134238] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3605.993576] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 18:03:45 (1778364225) [ 3630.337358] LustreError: 6483:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f0765886300 x1864746310883456/t0(0) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:266/0 lens 664/0 e 0 to 0 dl 1778364261 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3630.352141] LustreError: 6483:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 3641.383170] LustreError: 6483:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3641.395736] LustreError: 6486:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f077d4cce00 x1864746310884480/t0(0) o35->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:277/0 lens 392/0 e 0 to 0 dl 1778364272 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3646.433232] LustreError: 145297:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f075dc20e00 x1864746314526080/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:282/0 lens 544/0 e 0 to 0 dl 1778364277 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 3646.444477] LustreError: 145297:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 56 previous similar messages [ 3653.185258] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 18:04:33 (1778364273) [ 3677.230601] LustreError: 41052:0:(tgt_handler.c:2787:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 3688.255128] LustreError: 41052:0:(tgt_handler.c:2787:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 3691.528456] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 18:05:11 (1778364311) [ 3714.657987] LustreError: 8392:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f0779c77800 x1864746310907008/t0(0) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:350/0 lens 576/0 e 0 to 0 dl 1778364345 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3714.668324] LustreError: 8392:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 3719.767183] LustreError: 8392:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3719.771867] LustreError: 6483:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f07817ed180 x1864746310907776/t0(0) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:355/0 lens 664/0 e 0 to 0 dl 1778364350 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3730.615126] LustreError: 26179:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3730.620592] LustreError: 6485:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f077a340e00 x1864746310929280/t0(0) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:405/0 lens 664/0 e 0 to 0 dl 1778364400 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3730.629805] LustreError: 6485:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 119 previous similar messages [ 3742.485517] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 18:06:02 (1778364362) [ 3827.344989] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 18:07:27 (1778364447) [ 3850.120479] LustreError: 8392:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f064b589500 x1864746310982400/t0(0) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:485/0 lens 576/0 e 0 to 0 dl 1778364480 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 3850.126699] LustreError: 8392:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 98 previous similar messages [ 3850.128933] LustreError: 8392:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3850.131250] LustreError: 8392:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 3850.551081] LustreError: 8392:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3866.377060] LustreError: 6485:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 3866.379603] LustreError: 6485:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3866.799078] LustreError: 6485:0:(service.c:2538:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 3866.802922] LustreError: 6485:0:(service.c:2538:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 3882.225719] LustreError: 6483:0:(service.c:2537:ptlrpc_server_handle_request()) @@@ HIT req@ffff9f064677ca80 x1864746311000832/t0(0) o36->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:518/0 lens 504/0 e 0 to 0 dl 1778364513 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 3882.234380] LustreError: 6483:0:(service.c:2537:ptlrpc_server_handle_request()) Skipped 80 previous similar messages [ 3894.385191] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 18:08:34 (1778364514) [ 3919.091506] Lustre: DEBUG MARKER: phase 2 [ 3921.988989] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 18:09:02 (1778364542) [ 3992.114050] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 18:10:12 (1778364612) [ 3992.590229] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 3993.110203] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 18:10:13 (1778364613) [ 3994.849475] Lustre: DEBUG MARKER: Started rundbench load pid=134082 ... [ 3998.447374] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3999.981198] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 4000.630062] Lustre: Failing over lustre-MDT0000 [ 4000.952416] Lustre: server umount lustre-MDT0000 complete [ 4001.751836] LustreError: 8392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4001.758048] LustreError: 8392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 84 previous similar messages [ 4004.831338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4004.834314] LustreError: Skipped 9 previous similar messages [ 4004.836256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.840499] Lustre: Skipped 20 previous similar messages [ 4015.091868] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4015.093865] LDISKFS-fs (dm-0): recovery complete [ 4015.097588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4015.138418] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4015.141867] LustreError: Skipped 8 previous similar messages [ 4015.221265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.223259] Lustre: Skipped 7 previous similar messages [ 4015.236548] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4015.239319] Lustre: Skipped 10 previous similar messages [ 4015.756327] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4015.760059] Lustre: Skipped 11 previous similar messages [ 4016.558849] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4020.709207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4020.713118] Lustre: Skipped 21 previous similar messages [ 4020.718894] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4020.722374] Lustre: Skipped 11 previous similar messages [ 4020.740029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5089) [ 4020.740288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5025) [ 4021.906171] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4022.473212] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4027.299707] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4028.823397] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 4029.476895] Lustre: Failing over lustre-MDT0001 [ 4029.480928] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 4029.706242] Lustre: server umount lustre-MDT0001 complete [ 4043.652988] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 4043.654792] LDISKFS-fs (dm-1): recovery complete [ 4043.658980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4044.978900] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4049.276148] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:834 to 0x280000400:865) [ 4049.276189] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:817 to 0x2c0000400:833) [ 4050.441078] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4050.983232] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4055.836688] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4057.350570] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 4057.980893] Lustre: Failing over lustre-MDT0000 [ 4058.160966] Lustre: server umount lustre-MDT0000 complete [ 4072.237599] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4072.239918] LDISKFS-fs (dm-0): recovery complete [ 4072.243101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4073.777557] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4077.563995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5121) [ 4077.564057] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5057) [ 4078.760194] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4079.295056] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4084.164257] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4085.685223] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 4086.305137] Lustre: Failing over lustre-MDT0001 [ 4086.316474] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 4086.594790] Lustre: server umount lustre-MDT0001 complete [ 4100.568052] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 4100.569352] LDISKFS-fs (dm-1): recovery complete [ 4100.572753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.929006] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4106.085500] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1284 to 0x280000400:1313) [ 4106.085500] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1252 to 0x2c0000400:1281) [ 4107.284646] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4107.829751] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4112.771925] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4114.282624] Lustre: DEBUG MARKER: test_70b fail mds1 5 times [ 4114.911870] Lustre: Failing over lustre-MDT0000 [ 4115.191369] Lustre: server umount lustre-MDT0000 complete [ 4129.506283] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4129.509748] LDISKFS-fs (dm-0): recovery complete [ 4129.515016] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.992481] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4134.939105] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5010 to 0x2c0000401:5089) [ 4134.939143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5047 to 0x280000401:5153) [ 4136.468078] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4137.149279] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4141.031556] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 18:12:40 (1778364760) [ 4264.286200] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4275.199048] Lustre: DEBUG MARKER: test_70c fail mds1 1 times [ 4275.796863] Lustre: Failing over lustre-MDT0000 [ 4276.096796] Lustre: server umount lustre-MDT0000 complete [ 4290.724820] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 4290.726321] LDISKFS-fs (dm-0): recovery complete [ 4290.729440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.326463] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4299.057272] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:6278 to 0x2c0000401:6305) [ 4299.058494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:6342 to 0x280000401:6369) [ 4300.292853] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4301.028432] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4425.822198] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4436.591337] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 4437.342406] Lustre: Failing over lustre-MDT0001 [ 4437.361395] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 4437.604551] Lustre: server umount lustre-MDT0001 complete [ 4452.257127] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4452.259774] LDISKFS-fs (dm-1): recovery complete [ 4452.266972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4453.940380] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4460.548593] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3645 to 0x2c0000400:3681) [ 4460.551949] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3678 to 0x280000400:3713) [ 4461.793709] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4462.330075] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4481.374584] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 18:18:21 (1778365101) [ 4604.821373] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4615.545828] Lustre: DEBUG MARKER: test_70d fail mds2 1 times [ 4616.239862] Lustre: Failing over lustre-MDT0001 [ 4616.245572] LustreError: lustre-MDT0001-osp-MDT0000: operation out_update to node 0@lo failed: rc = -19 [ 4616.248688] LustreError: Skipped 5 previous similar messages [ 4616.250556] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4616.255411] Lustre: Skipped 24 previous similar messages [ 4616.257651] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4621.280852] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4621.286041] Lustre: Skipped 3 previous similar messages [ 4621.900947] Lustre: server umount lustre-MDT0001 complete [ 4622.989233] LustreError: 6483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4623.000299] LustreError: 6483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 94 previous similar messages [ 4637.405157] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4637.406724] LDISKFS-fs (dm-1): recovery complete [ 4637.412199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.523565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4637.525904] Lustre: Skipped 6 previous similar messages [ 4637.533761] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 4637.535672] Lustre: Skipped 6 previous similar messages [ 4638.349351] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4638.355429] Lustre: Skipped 6 previous similar messages [ 4639.118384] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4642.788283] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4642.792660] Lustre: Skipped 24 previous similar messages [ 4644.342440] Lustre: lustre-MDT0001: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 4644.347092] Lustre: Skipped 6 previous similar messages [ 4644.368314] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4041 to 0x2c0000400:4065) [ 4644.369352] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4073 to 0x280000400:4097) [ 4645.937650] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4646.638277] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4771.817948] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4782.854512] Lustre: DEBUG MARKER: test_70d fail mds2 2 times [ 4783.803477] Lustre: Failing over lustre-MDT0001 [ 4783.821247] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 4789.886494] Lustre: server umount lustre-MDT0001 complete [ 4805.169080] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4805.171392] LDISKFS-fs (dm-1): recovery complete [ 4805.176395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4805.295783] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4806.789077] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4812.193694] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4401 to 0x280000400:4417) [ 4812.193764] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4369 to 0x2c0000400:4385) [ 4813.774119] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4814.301714] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4818.286359] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 18:23:58 (1778365438) [ 4942.563221] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4953.595965] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 4954.605429] Lustre: Failing over lustre-MDT0001 [ 4954.871558] Lustre: server umount lustre-MDT0001 complete [ 4970.260966] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 4970.264043] LDISKFS-fs (dm-1): recovery complete [ 4970.273072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.460699] Lustre: lustre-MDT0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 4972.067501] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4975.610203] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4427 to 0x280000400:4449) [ 4975.610243] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4394 to 0x2c0000400:4417) [ 4976.823443] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4977.484652] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5102.808703] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5113.502755] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 5114.070031] Lustre: Failing over lustre-MDT0000 [ 5114.081479] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 5114.083393] Lustre: Skipped 4 previous similar messages [ 5114.277227] Lustre: server umount lustre-MDT0000 complete [ 5127.818065] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5127.819329] LDISKFS-fs (dm-0): recovery complete [ 5127.822479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.859785] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5127.862539] LustreError: Skipped 3 previous similar messages [ 5129.024575] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5136.146506] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11270 to 0x2c0000401:11297) [ 5136.150417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11334 to 0x280000401:11361) [ 5137.400026] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5138.077440] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5141.736617] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 18:29:21 (1778365761) [ 5147.371419] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5148.899580] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5149.490368] Lustre: Failing over lustre-OST0000 [ 5149.533020] Lustre: server umount lustre-OST0000 complete [ 5163.885595] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5163.887900] LDISKFS-fs (dm-2): recovery complete [ 5163.892762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5165.770189] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5169.138254] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5169.713121] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5179.580677] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5181.031566] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5181.602333] Lustre: Failing over lustre-OST0000 [ 5181.645507] Lustre: server umount lustre-OST0000 complete [ 5196.968201] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5196.970214] LDISKFS-fs (dm-2): recovery complete [ 5196.974626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5199.381371] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5202.725948] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5203.360705] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5209.417331] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 18:30:29 (1778365829) [ 5332.658955] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5335.846477] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5346.757436] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 5347.667303] Lustre: Failing over lustre-MDT0000 [ 5347.721280] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 5347.980881] Lustre: server umount lustre-MDT0000 complete [ 5349.475267] LustreError: 6468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778365969 with bad export cookie 10221156371495910880 [ 5349.478297] Lustre: Failing over lustre-MDT0001 [ 5349.482848] LustreError: 6468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5349.661682] Lustre: server umount lustre-MDT0001 complete [ 5365.663951] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 5365.665795] LDISKFS-fs (dm-0): recovery complete [ 5365.669615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.689085] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5365.692190] LDISKFS-fs (dm-1): recovery complete [ 5365.697562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5366.239202] Lustre: 3631:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778365970/real 1778365970] req@ffff9f077cdddf80 x1864746344107392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778365986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5366.239268] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.254570] Lustre: 3631:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 5366.261335] Lustre: Skipped 19 previous similar messages [ 5370.335232] Lustre: 3631:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778365974/real 1778365974] req@ffff9f077cddd500 x1864746344107776/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778365990 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5370.352496] Lustre: 3631:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 5374.431376] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f0649fa6d80 x1864746344110080/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5374.531368] LustreError: 8376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5374.536831] LustreError: 8376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 64 previous similar messages [ 5374.548508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5374.550735] Lustre: Skipped 3 previous similar messages [ 5374.561749] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5374.564680] Lustre: Skipped 5 previous similar messages [ 5374.629522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 5374.633076] LustreError: Skipped 8 previous similar messages [ 5375.564588] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5375.566856] Lustre: Skipped 5 previous similar messages [ 5375.583109] Lustre: 3633:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778365979/real 1778365979] req@ffff9f077cf79500 x1864746344108544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778365995 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5375.961233] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5376.066989] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5376.738504] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5376.742324] Lustre: Skipped 16 previous similar messages [ 5377.765316] Lustre: lustre-MDT0001: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5377.767755] Lustre: Skipped 5 previous similar messages [ 5377.780600] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4451 to 0x280000400:4481) [ 5377.780603] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4419 to 0x2c0000400:4449) [ 5381.855179] Lustre: 3633:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778365985/real 1778365985] req@ffff9f077cf7a300 x1864746344108800/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778366001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5381.873084] Lustre: 3633:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 5390.229339] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11369 to 0x2c0000401:11393) [ 5390.229344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11433 to 0x280000401:11457) [ 5391.595380] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5392.187626] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5392.705619] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5396.367900] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 18:33:36 (1778366016) [ 5399.483785] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5400.728857] Lustre: Failing over lustre-MDT0000 [ 5400.932240] Lustre: server umount lustre-MDT0000 complete [ 5416.419950] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5416.422621] LDISKFS-fs (dm-0): recovery complete [ 5416.426812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5417.796306] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5420.688432] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 5437.074787] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5437.084426] Lustre: 191363:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f064f6a6680 x1864746385479168/t339302417109(339302417109) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:562/0 lens 520/3488 e 0 to 0 dl 1778366067 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5437.109947] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11369 to 0x2c0000401:11425) [ 5437.109970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11489) [ 5438.578224] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5439.310380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5443.630883] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 18:34:23 (1778366063) [ 5446.219955] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5447.544800] Lustre: Failing over lustre-MDT0000 [ 5447.648243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5447.650080] Lustre: Skipped 3 previous similar messages [ 5447.836394] Lustre: server umount lustre-MDT0000 complete [ 5462.927575] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5462.930211] LDISKFS-fs (dm-0): recovery complete [ 5462.935374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.597167] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5466.716642] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 5466.719715] LustreError: 191362:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f064a3d4000 x1864746385479168/t339302417109(339302417109) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:592/0 lens 520/664 e 0 to 0 dl 1778366097 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5483.166660] Lustre: lustre-MDT0000: Client 6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4 (at 192.168.204.32@tcp) reconnected, waiting for 2 clients in recovery for 0:53 [ 5483.177636] Lustre: 191361:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f0649fa4000 x1864746385479168/t339302417109(339302417109) o101->6d39bb6b-b391-4ad0-93ef-7a10a7b2adf4@192.168.204.32@tcp:609/0 lens 520/3488 e 0 to 0 dl 1778366114 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 5483.230566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11521) [ 5483.230642] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11427 to 0x2c0000401:11457) [ 5485.038092] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5485.778353] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5489.482524] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 18:35:09 (1778366109) [ 5491.084672] Lustre: Failing over lustre-OST0000 [ 5491.142586] Lustre: server umount lustre-OST0000 complete [ 5492.925758] Lustre: Failing over lustre-MDT0000 [ 5493.039449] Lustre: server umount lustre-MDT0000 complete [ 5505.812543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5507.394238] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5508.438168] Lustre: lustre-MDT0000: Denying connection for new client 13340ae5-e737-4356-8d95-64d96ccf7bec (at 192.168.204.32@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 5511.164214] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11427 to 0x2c0000401:11489) [ 5516.162538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5517.746915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11553) [ 5518.260248] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5522.094686] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 18:35:42 (1778366142) [ 5522.401414] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5522.403744] LustreError: 8377:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f06495ec380 x1864746345441920/t356482285575(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:648/0 lens 1312/4320 e 0 to 0 dl 1778366153 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5525.054196] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5525.829352] Lustre: Failing over lustre-MDT0000 [ 5526.192257] Lustre: server umount lustre-MDT0000 complete [ 5538.783196] Lustre: 191418:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778366142/real 1778366142] req@ffff9f077a77f100 x1864746345441920/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1778366158 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5538.799357] Lustre: 191418:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 5541.297879] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5541.299104] LDISKFS-fs (dm-0): recovery complete [ 5541.302036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.499267] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5546.482797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11585) [ 5546.482835] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11521) [ 5547.556189] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5548.073684] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5552.056413] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 18:36:12 (1778366172) [ 5555.138500] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5558.224217] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5559.047963] Lustre: Failing over lustre-MDT0001 [ 5565.046348] Lustre: server umount lustre-MDT0001 complete [ 5579.120986] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5579.122544] LDISKFS-fs (dm-1): recovery complete [ 5579.128217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.339354] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5584.383634] Lustre: 195963:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f064a544a80 x1864746390779520/t34359740447(0) o36->13340ae5-e737-4356-8d95-64d96ccf7bec@192.168.204.32@tcp:734/0 lens 560/2880 e 0 to 0 dl 1778366239 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5584.396710] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4460 to 0x2c0000400:4481) [ 5584.396724] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4492 to 0x280000400:4513) [ 5585.768349] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5586.409968] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5590.056267] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 18:36:49 (1778366209) [ 5590.531551] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5590.535366] Lustre: Skipped 1 previous similar message [ 5590.538590] LustreError: 174393:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f064b1a6d80 x1864746345503488/t360777252897(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:716/0 lens 2520/4320 e 0 to 0 dl 1778366221 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5590.553386] LustreError: 174393:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 5593.299306] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5596.480580] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5597.327076] Lustre: Failing over lustre-MDT0000 [ 5597.574170] Lustre: server umount lustre-MDT0000 complete [ 5606.879145] Lustre: 201771:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778366210/real 1778366210] req@ffff9f064b1a4a80 x1864746345503488/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 2520/4320 e 0 to 1 dl 1778366226 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5612.112408] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5612.114108] LDISKFS-fs (dm-0): recovery complete [ 5612.117615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5613.471985] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5617.714233] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11553) [ 5617.714235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11617) [ 5618.761360] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5619.236179] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5621.361357] Lustre: Failing over lustre-MDT0001 [ 5621.390619] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 5621.394383] Lustre: Skipped 9 previous similar messages [ 5621.485519] Lustre: server umount lustre-MDT0001 complete [ 5636.459162] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5636.460353] LDISKFS-fs (dm-1): recovery complete [ 5636.463842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5637.757737] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5641.711746] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4524 to 0x280000400:4545) [ 5641.711752] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4492 to 0x2c0000400:4513) [ 5643.046801] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5643.855118] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5648.560797] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 18:37:48 (1778366268) [ 5655.712931] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5658.824941] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5659.372027] Lustre: Failing over lustre-MDT0000 [ 5659.667705] Lustre: server umount lustre-MDT0000 complete [ 5661.318544] LustreError: 145311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778366281 with bad export cookie 10221156371496721949 [ 5661.320644] Lustre: Failing over lustre-MDT0001 [ 5661.326449] LustreError: 145311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5666.783976] LustreError: 207083:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9f0779ff5f80 x1864746345567360/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 5666.802518] LustreError: 207083:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 5666.937827] Lustre: server umount lustre-MDT0001 complete [ 5681.573813] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5681.575872] LDISKFS-fs (dm-1): recovery complete [ 5681.582778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.583845] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5681.588512] LDISKFS-fs (dm-0): recovery complete [ 5681.591433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5681.658504] LustreError: 208178:0:(llog.c:1646:llog_backup()) MGC192.168.204.132@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 5681.660996] Lustre: 208178:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.132@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 5686.751381] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f077a2d3800 x1864746345568512/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5688.079035] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5688.190795] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5692.089317] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11649) [ 5692.089501] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11585) [ 5692.106902] Lustre: 208267:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f077cddd180 x1864746390848128/t42949673030(0) o36->13340ae5-e737-4356-8d95-64d96ccf7bec@192.168.204.32@tcp:88/0 lens 560/2880 e 0 to 0 dl 1778366348 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5692.113780] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4556 to 0x280000400:4577) [ 5692.113805] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4524 to 0x2c0000400:4545) [ 5693.175532] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5693.646962] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5694.088321] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5697.256104] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 18:38:37 (1778366317) [ 5697.552284] LustreError: 208189:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f077d289500 x1864746390881024/t47244640324(0) o36->13340ae5-e737-4356-8d95-64d96ccf7bec@192.168.204.32@tcp:68/0 lens 560/448 e 0 to 0 dl 1778366328 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5697.558204] LustreError: 208189:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 1 previous similar message [ 5702.873198] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5703.599676] Lustre: Failing over lustre-MDT0000 [ 5703.945175] Lustre: server umount lustre-MDT0000 complete [ 5714.060778] Lustre: lustre-MDT0001: Client 13340ae5-e737-4356-8d95-64d96ccf7bec (at 192.168.204.32@tcp) reconnecting [ 5714.064376] Lustre: Skipped 2 previous similar messages [ 5717.954429] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5717.955735] LDISKFS-fs (dm-0): recovery complete [ 5717.958455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5719.322331] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5723.146772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11681) [ 5723.146845] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11617) [ 5724.201992] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5724.823229] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5728.721627] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 18:39:08 (1778366348) [ 5729.046021] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 5729.047423] Lustre: Skipped 2 previous similar messages [ 5731.398302] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5732.091782] Lustre: Failing over lustre-MDT0001 [ 5732.183393] Lustre: server umount lustre-MDT0001 complete [ 5746.877044] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5746.878208] LDISKFS-fs (dm-1): recovery complete [ 5746.882387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5748.050391] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5752.310579] Lustre: 208267:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f077b4ddc00 x1864746390911616/t47244640395(0) o36->13340ae5-e737-4356-8d95-64d96ccf7bec@192.168.204.32@tcp:123/0 lens 560/2880 e 0 to 0 dl 1778366383 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5752.316588] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4566 to 0x2c0000400:4609) [ 5752.316592] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4598 to 0x280000400:4641) [ 5752.320831] Lustre: 208267:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5753.820881] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5754.585639] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5758.011687] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 18:39:37 (1778366377) [ 5763.689653] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5766.975768] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5767.891642] Lustre: Failing over lustre-MDT0000 [ 5768.244833] Lustre: server umount lustre-MDT0000 complete [ 5783.034757] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5783.036439] LDISKFS-fs (dm-0): recovery complete [ 5783.039207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.084239] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5783.087189] LustreError: Skipped 8 previous similar messages [ 5784.438579] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5788.676646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11713) [ 5788.676651] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11649) [ 5790.070588] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5790.807950] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5792.728943] Lustre: Failing over lustre-MDT0001 [ 5792.810822] Lustre: server umount lustre-MDT0001 complete [ 5807.551969] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5807.553250] LDISKFS-fs (dm-1): recovery complete [ 5807.557036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.749744] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5812.723550] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4620 to 0x2c0000400:4641) [ 5812.723563] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4652 to 0x280000400:4673) [ 5813.757335] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5814.258860] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5818.354648] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 18:40:38 (1778366438) [ 5824.362894] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5827.552445] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5828.169682] Lustre: Failing over lustre-MDT0000 [ 5828.396321] Lustre: server umount lustre-MDT0000 complete [ 5829.454567] LustreError: 42102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778366449 with bad export cookie 10221156371496735683 [ 5829.455885] Lustre: Failing over lustre-MDT0001 [ 5829.457759] LustreError: 42102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5829.546857] Lustre: server umount lustre-MDT0001 complete [ 5845.339037] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5845.342165] LDISKFS-fs (dm-0): recovery complete [ 5845.344835] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5845.346908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5845.347813] LDISKFS-fs (dm-1): recovery complete [ 5845.358499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.503142] Lustre: 3631:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778366453/real 1778366453] req@ffff9f0781419500 x1864746345721088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778366469 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5854.623623] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f06497a7480 x1864746345724032/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5856.197896] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5856.290302] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5861.014877] Lustre: 218796:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f06456bbb80 x1864746390978048/t55834574918(0) o36->13340ae5-e737-4356-8d95-64d96ccf7bec@192.168.204.32@tcp:231/0 lens 560/2880 e 0 to 0 dl 1778366491 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 5861.021122] Lustre: 218796:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 5861.021514] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4684 to 0x280000400:4705) [ 5861.022098] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4652 to 0x2c0000400:4673) [ 5869.238098] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11681) [ 5869.238112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11745) [ 5870.884731] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5871.589290] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5872.229635] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5876.341331] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 18:41:36 (1778366496) [ 5876.815555] LustreError: 174393:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f064a872d80 x1864746345756288/t382252089357(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:247/0 lens 1728/4320 e 0 to 0 dl 1778366507 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5876.824286] LustreError: 174393:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 3 previous similar messages [ 5880.142327] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5880.915650] Lustre: Failing over lustre-MDT0001 [ 5880.922769] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 5880.925580] Lustre: Skipped 6 previous similar messages [ 5887.063821] Lustre: server umount lustre-MDT0001 complete [ 5901.935978] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5901.938281] LDISKFS-fs (dm-1): recovery complete [ 5901.943441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5903.337289] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5907.458447] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4684 to 0x2c0000400:4705) [ 5907.458473] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4716 to 0x280000400:4737) [ 5908.922445] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5909.596062] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5913.485219] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 18:42:13 (1778366533) [ 5916.934365] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5917.808205] Lustre: Failing over lustre-MDT0000 [ 5918.049733] Lustre: server umount lustre-MDT0000 complete [ 5929.439116] Lustre: 221108:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778366533/real 1778366533] req@ffff9f077a59bb80 x1864746345787904/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1488/4320 e 0 to 1 dl 1778366549 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 5929.456437] Lustre: 221108:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [ 5933.012097] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5933.013401] LDISKFS-fs (dm-0): recovery complete [ 5933.016164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5934.218529] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5938.168653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11777) [ 5938.168662] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11713) [ 5939.403001] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5940.008105] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5943.098221] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 18:42:43 (1778366563) [ 5945.550929] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5947.709509] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5948.246890] Lustre: Failing over lustre-MDT0000 [ 5948.580660] Lustre: server umount lustre-MDT0000 complete [ 5962.058853] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 5962.060030] LDISKFS-fs (dm-0): recovery complete [ 5962.062699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5963.267121] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5967.355466] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11745) [ 5967.355505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11809) [ 5968.372986] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5968.830375] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5970.636801] Lustre: Failing over lustre-MDT0001 [ 5970.715604] Lustre: server umount lustre-MDT0001 complete [ 5972.447909] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5972.451070] Lustre: Skipped 61 previous similar messages [ 5974.669546] LustreError: 218797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5974.674047] LustreError: 218797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 201 previous similar messages [ 5984.288183] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5984.289485] LDISKFS-fs (dm-1): recovery complete [ 5984.293027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5984.390741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5984.393142] Lustre: Skipped 20 previous similar messages [ 5984.400361] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5984.402490] Lustre: Skipped 20 previous similar messages [ 5984.908503] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5984.910752] Lustre: Skipped 20 previous similar messages [ 5985.499932] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5989.863400] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 5989.866334] Lustre: Skipped 69 previous similar messages [ 5989.871275] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5989.875545] Lustre: Skipped 20 previous similar messages [ 5989.889167] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4716 to 0x280000400:4769) [ 5989.889183] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4684 to 0x2c0000400:4737) [ 5990.970987] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5991.522722] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5995.963670] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 18:43:35 (1778366615) [ 5996.306601] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 5996.308135] Lustre: Skipped 5 previous similar messages [ 5998.481853] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6000.650790] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6001.218401] Lustre: Failing over lustre-MDT0000 [ 6001.562450] Lustre: server umount lustre-MDT0000 complete [ 6002.639908] LustreError: 42102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778366622 with bad export cookie 10221156371496746127 [ 6002.641160] Lustre: Failing over lustre-MDT0001 [ 6002.644110] LustreError: 42102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6007.775687] LustreError: 228299:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9f077d28b800 x1864746345864704/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6007.781368] LustreError: 228299:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6007.868973] Lustre: server umount lustre-MDT0001 complete [ 6022.041288] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6022.043262] LDISKFS-fs (dm-0): recovery complete [ 6022.043362] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6022.046179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.046972] LDISKFS-fs (dm-1): recovery complete [ 6022.053675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6022.125283] LustreError: 229397:0:(llog.c:1646:llog_backup()) MGC192.168.204.132@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6022.127946] Lustre: 229397:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.132@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6027.743353] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f076daf9180 x1864746345865856/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6027.883026] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_connect to node 0@lo failed: rc = -114 [ 6027.886643] LustreError: Skipped 16 previous similar messages [ 6029.085295] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6029.170634] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6033.975451] Lustre: 229410:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f064a872a00 x1864746391055360/t68719476738(0) o36->13340ae5-e737-4356-8d95-64d96ccf7bec@192.168.204.32@tcp:404/0 lens 496/2888 e 0 to 0 dl 1778366664 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6033.978349] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4716 to 0x280000400:4801) [ 6033.983947] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4684 to 0x2c0000400:4769) [ 6033.986081] Lustre: 229410:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6034.009061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11841) [ 6034.009081] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11777) [ 6035.055550] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6035.506184] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6035.933326] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6038.978156] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 18:44:19 (1778366659) [ 6042.440784] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6042.983875] Lustre: Failing over lustre-MDT0000 [ 6043.208489] Lustre: server umount lustre-MDT0000 complete [ 6054.540446] Lustre: lustre-MDT0001: Client 13340ae5-e737-4356-8d95-64d96ccf7bec (at 192.168.204.32@tcp) reconnecting [ 6054.543137] Lustre: Skipped 1 previous similar message [ 6056.889115] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6056.890908] LDISKFS-fs (dm-0): recovery complete [ 6056.894120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6058.102733] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6062.070829] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11809) [ 6062.070842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11873) [ 6063.084602] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6063.560390] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6066.693800] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 18:44:46 (1778366686) [ 6069.225654] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6069.775256] Lustre: Failing over lustre-MDT0001 [ 6069.848342] Lustre: server umount lustre-MDT0001 complete [ 6083.350569] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6083.351728] LDISKFS-fs (dm-1): recovery complete [ 6083.355718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6084.602423] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6088.705917] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4684 to 0x2c0000400:4801) [ 6088.705919] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4716 to 0x280000400:4833) [ 6089.826347] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6090.354906] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6093.643321] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 18:45:13 (1778366713) [ 6096.157410] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6098.332358] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6098.882801] Lustre: Failing over lustre-MDT0000 [ 6099.045983] Lustre: server umount lustre-MDT0000 complete [ 6112.647174] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6112.648297] LDISKFS-fs (dm-0): recovery complete [ 6112.650881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6113.972248] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6117.882346] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11841) [ 6117.882371] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11905) [ 6119.083320] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6119.585561] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6121.472812] Lustre: Failing over lustre-MDT0001 [ 6121.554239] Lustre: server umount lustre-MDT0001 complete [ 6135.369216] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6135.370474] LDISKFS-fs (dm-1): recovery complete [ 6135.374032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6136.726856] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6140.913313] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4684 to 0x2c0000400:4833) [ 6140.913349] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4716 to 0x280000400:4865) [ 6141.945609] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6142.438722] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6145.671629] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 18:46:05 (1778366765) [ 6145.989982] LustreError: 233116:0:(ldlm_lib.c:3326:target_send_reply_msg()) @@@ dropping reply req@ffff9f06456bb100 x1864746391108224/t81604378626(0) o36->13340ae5-e737-4356-8d95-64d96ccf7bec@192.168.204.32@tcp:516/0 lens 496/456 e 0 to 0 dl 1778366776 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6145.995893] LustreError: 233116:0:(ldlm_lib.c:3326:target_send_reply_msg()) Skipped 6 previous similar messages [ 6148.283081] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6150.749043] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6151.353129] Lustre: Failing over lustre-MDT0000 [ 6151.577386] Lustre: server umount lustre-MDT0000 complete [ 6152.726593] LustreError: 145322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778366772 with bad export cookie 10221156371496753666 [ 6152.727895] Lustre: Failing over lustre-MDT0001 [ 6152.729840] LustreError: 145322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6152.818787] Lustre: server umount lustre-MDT0001 complete [ 6167.258114] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6167.259972] LDISKFS-fs (dm-0): recovery complete [ 6167.263514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.271084] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6167.272461] LDISKFS-fs (dm-1): recovery complete [ 6167.277304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.639132] Lustre: 3631:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778366776/real 1778366776] req@ffff9f064a1bdf80 x1864746345991168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1778366792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6172.645978] Lustre: 3631:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6177.759415] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f064a1bdc00 x1864746345993856/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6179.111827] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6179.211744] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6182.054130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11459 to 0x280000401:11937) [ 6182.054139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11491 to 0x2c0000401:11873) [ 6187.166093] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4716 to 0x280000400:4897) [ 6187.166109] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4684 to 0x2c0000400:4865) [ 6188.333948] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6188.997153] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6189.545738] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6192.990595] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 18:46:53 (1778366813) [ 6193.574862] Lustre: 241274:0:(genops.c:1793:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 13340ae5-e737-4356-8d95-64d96ccf7bec at adminstrative request [ 6198.441352] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 18:46:58 (1778366818) [ 6200.409568] Lustre: Failing over lustre-MDT0000 [ 6200.571594] Lustre: server umount lustre-MDT0000 complete [ 6213.369364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6214.622342] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6218.755460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11988 to 0x280000401:12033) [ 6218.755498] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11925 to 0x2c0000401:11969) [ 6219.892739] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6220.489032] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6224.054493] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 18:47:24 (1778366844) [ 6227.781656] Lustre: Failing over lustre-OST0000 [ 6227.842928] Lustre: server umount lustre-OST0000 complete [ 6240.626982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6242.455106] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6245.407791] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6245.898907] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6249.561136] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 18:47:49 (1778366869) [ 6250.848485] Lustre: Failing over lustre-MDT0000 [ 6251.051099] Lustre: server umount lustre-MDT0000 complete [ 6253.528041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.901131] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6255.701073] Lustre: lustre-MDT0000: Denying connection for new client 4dd6b4c3-a362-4bc3-aa75-969f6e85c42d (at 192.168.204.32@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 6255.711560] Lustre: Skipped 1 previous similar message [ 6258.676898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:11988 to 0x280000401:12065) [ 6258.676899] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11925 to 0x2c0000401:12001) [ 6263.025504] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 18:48:02 (1778366882) [ 6265.989594] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6266.834968] Lustre: Failing over lustre-OST0000 [ 6266.878476] Lustre: server umount lustre-OST0000 complete [ 6280.815846] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6280.817873] LDISKFS-fs (dm-2): recovery complete [ 6280.821725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6282.751488] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6286.317513] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6287.006514] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6290.570934] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 18:48:30 (1778366910) [ 6293.579918] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6295.394788] Lustre: Failing over lustre-OST0000 [ 6295.426795] Lustre: server umount lustre-OST0000 complete [ 6309.525532] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6309.529473] LDISKFS-fs (dm-2): recovery complete [ 6309.536336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6310.842896] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.204.32@tcp inode [0x20002f9d1:0x5:0x0] object 0x280000401:12067 extent [0-4194303]: client csum 172e53d6, server csum 33793b2a [ 6311.507413] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6314.564704] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6315.087366] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6318.606319] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 18:48:58 (1778366938) [ 6321.071860] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 6323.640616] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6325.595555] Lustre: Failing over lustre-MDT0000 [ 6325.861504] Lustre: server umount lustre-MDT0000 complete [ 6327.134692] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778366946 with bad export cookie 10221156371496779790 [ 6327.136811] Lustre: Failing over lustre-OST0000 [ 6327.138186] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6327.181109] Lustre: server umount lustre-OST0000 complete [ 6341.394090] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6341.395547] LDISKFS-fs (dm-0): recovery complete [ 6341.398524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.839731] LustreError: 3630:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9f0781c6d880 x1864746346139392/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6353.087610] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6357.061184] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11925 to 0x2c0000401:12033) [ 6367.059299] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 6367.062353] LDISKFS-fs (dm-2): recovery complete [ 6367.068742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6368.788799] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6374.222960] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 18:49:54 (1778366994) [ 6378.769969] Lustre: Failing over lustre-OST0000 [ 6378.828224] Lustre: server umount lustre-OST0000 complete [ 6380.020288] Lustre: Failing over lustre-MDT0000 [ 6380.252606] Lustre: server umount lustre-MDT0000 complete [ 6392.618329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.654551] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6392.657372] LustreError: Skipped 10 previous similar messages [ 6393.788894] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6397.946336] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:11925 to 0x2c0000401:12065) [ 6404.237099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.035478] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6406.713648] Lustre: lustre-OST0000: Denying connection for new client 85c4b25a-e707-4423-9040-a27ef1e113c0 (at 192.168.204.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 6406.721022] Lustre: Skipped 1 previous similar message [ 6411.916857] Lustre: lustre-OST0000: Denying connection for new client 85c4b25a-e707-4423-9040-a27ef1e113c0 (at 192.168.204.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:03 [ 6411.922234] Lustre: Skipped 1 previous similar message [ 6422.156973] Lustre: lustre-OST0000: Denying connection for new client 85c4b25a-e707-4423-9040-a27ef1e113c0 (at 192.168.204.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 6422.161110] Lustre: Skipped 1 previous similar message [ 6442.636771] Lustre: lustre-OST0000: Denying connection for new client 85c4b25a-e707-4423-9040-a27ef1e113c0 (at 192.168.204.32@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:32 [ 6442.641872] Lustre: Skipped 3 previous similar messages [ 6475.500086] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 6475.504280] Lustre: 254330:0:(genops.c:1620:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 4dd6b4c3-a362-4bc3-aa75-969f6e85c42d@ [ 6475.511181] Lustre: 254330:0:(genops.c:1620:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 6475.516088] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 6475.527523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12116 to 0x280000401:12137) [ 6479.352559] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 6483.855296] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 6486.048163] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 18:51:46 (1778367106) [ 6487.404598] Lustre: Failing over lustre-OST0000 [ 6487.444811] Lustre: server umount lustre-OST0000 complete [ 6501.391087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6503.604771] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6507.896024] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 18:52:07 (1778367127) [ 6509.544971] Lustre: Failing over lustre-OST0000 [ 6509.611885] Lustre: server umount lustre-OST0000 complete [ 6522.446653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6523.884235] LustreError: 257307:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 6523.886713] LustreError: 257307:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 76 previous similar messages [ 6524.223619] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6530.015081] Lustre: *** cfs_fail_loc=715, val=40*** [ 6539.231110] Lustre: 3630:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1778367143/real 1778367143] req@ffff9f064af61180 x1864746346246912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1778367159 ref 1 fl Rpc:XQr/2c0/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 6539.237453] Lustre: 3630:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 18 previous similar messages [ 6539.239677] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:25 [ 6539.242380] Lustre: Skipped 1 previous similar message [ 6539.932866] Lustre: lustre-OST0000: Client 85c4b25a-e707-4423-9040-a27ef1e113c0 (at 192.168.204.32@tcp) reconnected, waiting for 3 clients in recovery for 1:24 [ 6545.375135] Lustre: *** cfs_fail_loc=715, val=40*** [ 6545.376649] Lustre: Skipped 2 previous similar messages [ 6555.615504] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 6555.617972] Lustre: Skipped 1 previous similar message [ 6561.759336] Lustre: *** cfs_fail_loc=715, val=40*** [ 6561.761859] Lustre: Skipped 2 previous similar messages [ 6563.927137] LustreError: 257307:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6563.932020] LustreError: 257307:0:(ldlm_lib.c:2885:target_recovery_thread()) Skipped 77 previous similar messages [ 6565.270576] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6565.764433] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6569.185881] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 18:53:09 (1778367189) [ 6570.484080] Lustre: Failing over lustre-MDT0000 [ 6570.834605] Lustre: server umount lustre-MDT0000 complete [ 6576.095785] LustreError: 240051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6576.100229] LustreError: 240051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 194 previous similar messages [ 6583.730050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.448158] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6587.020662] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6587.025287] Lustre: Skipped 19 previous similar messages [ 6588.903884] LustreError: 258753:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 6593.503226] Lustre: *** cfs_fail_loc=715, val=80*** [ 6593.506749] Lustre: Skipped 2 previous similar messages [ 6603.404503] Lustre: lustre-MDT0000: Client 85c4b25a-e707-4423-9040-a27ef1e113c0 (at 192.168.204.32@tcp) reconnected, waiting for 2 clients in recovery for 1:26 [ 6603.407935] Lustre: Skipped 1 previous similar message [ 6604.256314] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6609.887148] Lustre: *** cfs_fail_loc=715, val=80*** [ 6609.888718] Lustre: Skipped 1 previous similar message [ 6618.764674] Lustre: lustre-MDT0000: Client 85c4b25a-e707-4423-9040-a27ef1e113c0 (at 192.168.204.32@tcp) reconnected, waiting for 2 clients in recovery for 1:10 [ 6620.639427] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6625.247147] Lustre: *** cfs_fail_loc=715, val=80*** [ 6625.249213] Lustre: Skipped 1 previous similar message [ 6635.148673] Lustre: lustre-MDT0000: Client 85c4b25a-e707-4423-9040-a27ef1e113c0 (at 192.168.204.32@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 6637.023432] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6641.631135] Lustre: *** cfs_fail_loc=715, val=80*** [ 6641.632280] Lustre: Skipped 1 previous similar message [ 6651.532469] Lustre: lustre-MDT0000: Client 85c4b25a-e707-4423-9040-a27ef1e113c0 (at 192.168.204.32@tcp) reconnected, waiting for 2 clients in recovery for 0:37 [ 6652.383622] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6668.767596] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 6668.991124] LustreError: 258753:0:(ldlm_lib.c:2885:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 6668.999570] Lustre: 258753:0:(ldlm_lib.c:2931:target_recovery_thread()) too long recovery - read logs [ 6668.999678] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6669.004653] LustreError: dumping log to /tmp/lustre-log.1778367288.258753 [ 6669.007280] Lustre: Skipped 59 previous similar messages [ 6669.069238] Lustre: lustre-MDT0000: Recovery over after 1:22, of 2 clients 2 recovered and 0 were evicted. [ 6669.073293] Lustre: Skipped 19 previous similar messages [ 6669.092319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12151 to 0x280000401:12169) [ 6669.092322] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12078 to 0x2c0000401:12097) [ 6670.628269] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6671.193602] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6674.874761] Lustre: DEBUG MARKER: == replay-single test complete, duration 6538 sec ======== 18:54:54 (1778367294) [ 6675.351573] Lustre: DEBUG MARKER: === replay-single: start cleanup 18:54:55 (1778367295) === [ 6677.646004] Lustre: DEBUG MARKER: === replay-single: finish cleanup 18:54:57 (1778367297) === [ 6678.471170] Lustre: Failing over lustre-MDT0000 [ 6678.706761] Lustre: server umount lustre-MDT0000 complete [ 6679.519463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6679.522087] LustreError: Skipped 13 previous similar messages [ 6679.523450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.527171] Lustre: Skipped 53 previous similar messages [ 6694.405870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.565984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6694.568290] Lustre: Skipped 20 previous similar messages [ 6694.585561] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6694.587632] Lustre: Skipped 20 previous similar messages [ 6695.861109] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6700.029362] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:12078 to 0x2c0000401:12129) [ 6700.029460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:12151 to 0x280000401:12201) [ 6701.268815] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6701.895420] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6710.241522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6710.245776] Lustre: Skipped 28 previous similar messages [ 6714.013374] Lustre: server umount lustre-MDT0000 complete [ 6717.328663] LustreError: 6469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1778367337 with bad export cookie 10221156371496812319 [ 6717.332852] LustreError: 6469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6717.430158] Lustre: server umount lustre-MDT0001 complete [ 6730.825268] Lustre: server umount lustre-OST0000 complete [ 6744.099323] Lustre: server umount lustre-OST0001 complete [ 6749.229690] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6750.112534] Key type lgssc unregistered [ 6750.229323] LNet: 263340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6750.231466] LNetError: 263340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6750.242261] LNet: Removed LNI 192.168.204.132@tcp [ 6750.525088] Key type .llcrypt unregistered [ 6750.526279] Key type ._llcrypt unregistered