[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495263432 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002359] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006886] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010152] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013076] *** VALIDATE selinux *** [ 0.022489] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027540] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029017] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031013] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032101] *** VALIDATE tmpfs *** [ 0.034374] *** VALIDATE proc *** [ 0.035227] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.038063] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041024] Spectre V2 : User space: Vulnerable [ 0.042007] Speculative Store Bypass: Vulnerable [ 0.046061] debug: unmapping init [mem 0xffffffffafe59000-0xffffffffafe60fff] [ 0.049267] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051016] ... version: 2 [ 0.051905] ... bit width: 48 [ 0.052008] ... generic registers: 4 [ 0.052968] ... value mask: 0000ffffffffffff [ 0.053012] ... max period: 00007fffffffffff [ 0.054014] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056286] rcu: Hierarchical SRCU implementation. [ 0.058371] smp: Bringing up secondary CPUs ... [ 0.059667] x86: Booting SMP configuration: [ 0.060031] .... node #0, CPUs: #1 #2 #3 [ 0.069118] smp: Brought up 1 node, 4 CPUs [ 0.071016] smpboot: Max logical packages: 1 [ 0.072055] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125615] node 0 deferred pages initialised in 51ms [ 0.126541] devtmpfs: initialized [ 0.127326] x86/mm: Memory block size: 128MB [ 0.131202] gcov: version magic: 0x41383552 [ 0.134286] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.140344] pinctrl core: initialized pinctrl subsystem [ 0.142239] [ 0.142923] ************************************************************* [ 0.146010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148009] ** ** [ 0.151014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155252] ** ** [ 0.159015] ** This means that this kernel is built to expose internal ** [ 0.162011] ** IOMMU data structures, which may compromise security on ** [ 0.165015] ** your system. ** [ 0.168011] ** ** [ 0.171012] ** If you see this message and you are not debugging the ** [ 0.174012] ** kernel, report this immediately to your vendor! ** [ 0.177013] ** ** [ 0.180021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183013] ************************************************************* [ 0.186690] NET: Registered protocol family 16 [ 0.188612] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195151] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200103] cpuidle: using governor menu [ 0.202000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204629] PCI: Using configuration type 1 for base access [ 0.207211] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215164] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.219450] cryptd: max_cpu_qlen set to 1000 [ 0.221233] ACPI: Added _OSI(Module Device) [ 0.223015] ACPI: Added _OSI(Processor Device) [ 0.225014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227079] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233759] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.241454] ACPI: Interpreter enabled [ 0.242048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244015] ACPI: Using IOAPIC for interrupt routing [ 0.246189] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249494] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.261909] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.264038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.268016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.272138] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.278636] acpiphp: Slot [2] registered [ 0.280110] acpiphp: Slot [5] registered [ 0.282181] acpiphp: Slot [6] registered [ 0.284429] acpiphp: Slot [7] registered [ 0.286275] acpiphp: Slot [8] registered [ 0.288112] acpiphp: Slot [9] registered [ 0.290162] acpiphp: Slot [10] registered [ 0.291189] acpiphp: Slot [3] registered [ 0.293074] acpiphp: Slot [4] registered [ 0.295081] acpiphp: Slot [11] registered [ 0.296074] acpiphp: Slot [12] registered [ 0.298158] acpiphp: Slot [13] registered [ 0.300084] acpiphp: Slot [14] registered [ 0.301083] acpiphp: Slot [15] registered [ 0.303161] acpiphp: Slot [16] registered [ 0.305160] acpiphp: Slot [17] registered [ 0.307155] acpiphp: Slot [18] registered [ 0.309078] acpiphp: Slot [19] registered [ 0.311094] acpiphp: Slot [20] registered [ 0.313159] acpiphp: Slot [21] registered [ 0.314064] acpiphp: Slot [22] registered [ 0.315340] acpiphp: Slot [23] registered [ 0.317073] acpiphp: Slot [24] registered [ 0.319271] acpiphp: Slot [25] registered [ 0.322207] acpiphp: Slot [26] registered [ 0.324135] acpiphp: Slot [27] registered [ 0.326096] acpiphp: Slot [28] registered [ 0.328078] acpiphp: Slot [29] registered [ 0.329143] acpiphp: Slot [30] registered [ 0.331261] acpiphp: Slot [31] registered [ 0.334075] PCI host bridge to bus 0000:00 [ 0.336018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.339118] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.342025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.346021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.351038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.353162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.356135] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.357919] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.367015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.371039] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373011] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375009] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377388] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380162] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.382028] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.384623] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.399014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.403013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.407458] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.413016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.420017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.438016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.450375] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.458014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480011] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.491866] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.498017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.507012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.522013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.535090] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.550019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.570017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.607019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.626098] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.638016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.651024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.685015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.696666] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.719103] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.740015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.773014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.789262] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.793064] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.796598] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.801173] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.804391] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.812025] iommu: Default domain type: Passthrough [ 0.813368] SCSI subsystem initialized [ 0.814127] ACPI: bus type USB registered [ 0.816184] usbcore: registered new interface driver usbfs [ 0.819080] usbcore: registered new interface driver hub [ 0.821079] usbcore: registered new device driver usb [ 0.823166] pps_core: LinuxPPS API ver. 1 registered [ 0.826013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.830099] PTP clock support registered [ 0.832327] EDAC MC: Ver: 3.0.0 [ 0.834085] PCI: Using ACPI for IRQ routing [ 0.834892] NetLabel: Initializing [ 0.836013] NetLabel: domain hash size = 128 [ 0.839045] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.841205] NetLabel: unlabeled traffic allowed by default [ 0.844436] vgaarb: loaded [ 0.846693] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.849040] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.859498] clocksource: Switched to clocksource kvm-clock [ 0.969238] VFS: Disk quotas dquot_6.6.0 [ 0.970345] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.972606] *** VALIDATE ramfs *** [ 0.973690] *** VALIDATE hugetlbfs *** [ 0.976380] pnp: PnP ACPI init [ 0.979065] pnp: PnP ACPI: found 6 devices [ 0.997165] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.999732] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.001420] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.003095] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.004603] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.006280] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.008138] NET: Registered protocol family 2 [ 1.010299] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.016196] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.020869] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.025132] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.029160] TCP: Hash tables configured (established 65536 bind 65536) [ 1.033193] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.037744] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.041377] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.045777] NET: Registered protocol family 1 [ 1.049185] RPC: Registered named UNIX socket transport module. [ 1.053838] RPC: Registered udp transport module. [ 1.056608] RPC: Registered tcp transport module. [ 1.061249] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.067708] NET: Registered protocol family 44 [ 1.105850] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.114594] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.124784] pci 0000:00:00.0: quirk_passive_release+0x0/0x90 took 13185 usecs [ 1.134183] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.136010] PCI: CLS 0 bytes, default 64 [ 1.139907] Unpacking initramfs... [ 2.671978] debug: unmapping init [mem 0xffff9b3b7cc54000-0xffff9b3b7ffbffff] [ 2.676085] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.678444] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.681627] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.266320] Initialise system trusted keyrings [ 3.267511] Key type blacklist registered [ 3.268946] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.277996] zbud: loaded [ 3.280795] *** VALIDATE nfs *** [ 3.281704] *** VALIDATE nfs4 *** [ 3.282857] pstore: using deflate compression [ 3.285841] Platform Keyring initialized [ 3.401943] NET: Registered protocol family 38 [ 3.403313] Key type asymmetric registered [ 3.404226] Asymmetric key parser 'x509' registered [ 3.405271] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.406902] io scheduler mq-deadline registered [ 3.407664] io scheduler kyber registered [ 3.408709] io scheduler bfq registered [ 3.409864] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.411404] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.412852] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.414136] ACPI: Power Button [PWRF] [ 3.419632] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.429901] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.461872] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.476124] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.509845] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.539586] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.574311] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.580811] Non-volatile memory driver v1.3 [ 3.583818] Linux agpgart interface v0.103 [ 3.630562] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.633574] vda: detected capacity change from 0 to 74649600 [ 3.656628] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.658852] vdb: detected capacity change from 0 to 1073741824 [ 3.674281] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.676889] vdc: detected capacity change from 0 to 2621440000 [ 3.704096] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.706670] vdd: detected capacity change from 0 to 2621440000 [ 3.724941] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.727913] vde: detected capacity change from 0 to 4294967296 [ 3.740986] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.743938] vdf: detected capacity change from 0 to 4294967296 [ 3.753715] libphy: Fixed MDIO Bus: probed [ 3.769353] usbcore: registered new interface driver usbserial_generic [ 3.773635] usbserial: USB Serial support registered for generic [ 3.777655] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.785348] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.788354] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.792313] mousedev: PS/2 mouse device common for all mice [ 3.796928] rtc_cmos 00:05: RTC can wake from S4 [ 3.801933] rtc_cmos 00:05: registered as rtc0 [ 3.805305] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.808279] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.830097] intel_pstate: CPU model not supported [ 3.837557] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.838499] hid: raw HID events driver (C) Jiri Kosina [ 3.848303] usbcore: registered new interface driver usbhid [ 3.851488] usbhid: USB HID core driver [ 3.854342] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.855837] drop_monitor: Initializing network drop monitor service [ 3.861273] Initializing XFRM netlink socket [ 3.864619] NET: Registered protocol family 10 [ 3.869888] Segment Routing with IPv6 [ 3.873349] NET: Registered protocol family 17 [ 3.875891] mpls_gso: MPLS GSO support [ 3.883814] RAS: Correctable Errors collector initialized. [ 3.886697] AVX version of gcm_enc/dec engaged. [ 3.888330] AES CTR mode by8 optimization enabled [ 3.969919] sched_clock: Marking stable (3969902928, 0)->(4935295523, -965392595) [ 3.973283] registered taskstats version 1 [ 3.975194] Loading compiled-in X.509 certificates [ 3.976693] zswap: loaded using pool lzo/zbud [ 4.001511] Key type big_key registered [ 4.012680] Key type encrypted registered [ 4.014054] ima: No TPM chip found, activating TPM-bypass! [ 4.015856] ima: Allocated hash algorithm: sha1 [ 4.017672] ima: No architecture policies found [ 4.019371] evm: Initialising EVM extended attributes: [ 4.021199] evm: security.selinux [ 4.022255] evm: security.ima [ 4.023326] evm: security.capability [ 4.024672] evm: HMAC attrs: 0x1 [ 4.027404] rtc_cmos 00:05: setting system clock to 2026-07-15 05:21:21 UTC (1784092881) [ 4.033785] debug: unmapping init [mem 0xffffffffb0e03000-0xffffffffb0ffffff] [ 4.037968] debug: unmapping init [mem 0xffffffffafb82000-0xffffffffafe58fff] [ 4.045184] Write protecting the kernel read-only data: 28672k [ 4.048765] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 4.051284] debug: unmapping init [mem 0xffffffffaeb14000-0xffffffffaebfffff] [ 4.099211] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.109885] systemd[1]: Detected virtualization kvm. [ 4.111791] systemd[1]: Detected architecture x86-64. [ 4.113832] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.142549] systemd[1]: No hostname configured. [ 4.145400] systemd[1]: Set hostname to . [ 4.148609] random: systemd: uninitialized urandom read (16 bytes read) [ 4.151118] systemd[1]: Initializing machine ID from random generator. [ 4.238635] random: ln: uninitialized urandom read (6 bytes read) [ 4.358974] random: systemd: uninitialized urandom read (16 bytes read) [ 4.361905] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.366738] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.374196] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.281418] device-mapper: uevent: version 1.0.3 [ 5.285455] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Startin[ 6.552552] virtio_net virtio0 ens2: renamed from eth0 g dracut initqueue hook... [ 6.592034] random: fast init done [ 6.679771] scsi host0: ata_piix [ 6.763529] scsi host1: ata_piix [ 6.767298] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.769829] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.531456] random: crng init done [ 11.532644] random: 7 urandom warning(s) missed due to ratelimiting [ 11.534160] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.317429] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.946995] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.560872] SELinux: Disabled at runtime. [ 14.667226] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.676468] systemd[1]: Detected virtualization kvm. [ 14.678202] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.590327] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.595402] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.602780] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.611839] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.617164] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.624412] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.629985] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 15.827707] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.238575] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.651613] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.713779] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.853043] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.879773] EDAC sbridge: Ver: 1.1.2 [ 19.500833] Key type dns_resolver registered [ 19.890926] NFS: Registering the id_resolver key type [ 19.893276] Key type id_resolver registered [ 19.895100] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg112-server login: [ 50.639297] libcfs: loading out-of-tree module taints kernel. [ 50.661805] Key type ._llcrypt registered [ 50.663295] Key type .llcrypt registered [ 50.739303] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 60.171686] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 61.045991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.055472] alg: No test for adler32 (adler32-zlib) [ 62.206441] Lustre: Lustre: Build Version: 2.17.54_162_gc3b23b4 [ 62.658385] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 64.320147] Key type lgssc registered [ 65.189722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 74.707516] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.757449] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 102.422996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 102.435352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 103.540214] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 103.555535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 103.602340] Lustre: lustre-MDT0000: new disk, initializing [ 103.649745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 103.668352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 105.246840] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.845029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 111.902563] Lustre: 6491:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 111.917823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 111.920386] Lustre: Skipped 1 previous similar message [ 111.965378] Lustre: lustre-MDT0001: new disk, initializing [ 111.997036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 112.010797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 112.016162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 113.793812] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.556424] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 120.515032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 120.638871] Lustre: lustre-OST0000: new disk, initializing [ 120.640553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 120.643877] Lustre: 8427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 120.680708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 123.318579] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 125.961819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 125.970833] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 126.016980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 131.122747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 131.185359] Lustre: lustre-OST0001: new disk, initializing [ 131.187760] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 131.191839] Lustre: 9500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 131.220457] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 134.140638] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 134.842817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 134.847244] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 134.873440] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 140.934244] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 144.016033] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 146.642417] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing check_logdir /tmp/testlogs/ [ 148.570813] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing yml_node [ 150.486405] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 151.598160] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 152.753825] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 153.443454] Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Wed Jul 15 01:23:50 EDT 2026 [ 161.042741] Lustre: DEBUG MARKER: excepting tests: 59 36 [ 161.932234] Lustre: DEBUG MARKER: === replay-single: start setup 01:23:58 (1784093038) === [ 164.656469] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing check_config_client /mnt/lustre [ 174.429757] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 176.237270] Lustre: 13302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 178.144181] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.523284] Lustre: DEBUG MARKER: === replay-single: finish setup 01:24:17 (1784093057) === [ 181.432837] Lustre: DEBUG MARKER: == replay-single test 0a: empty replay =================== 01:24:18 (1784093058) [ 185.686497] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 186.867740] Lustre: Failing over lustre-MDT0000 [ 186.997669] Lustre: server umount lustre-MDT0000 complete [ 188.897409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 188.904885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 191.459819] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 191.468484] Lustre: Skipped 1 previous similar message [ 194.829728] LustreError: 6498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 194.839143] LustreError: 6498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 196.580736] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 196.594294] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 199.949972] LustreError: 6496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 203.861717] LDISKFS-fs (dm-0): 10 truncates cleaned up [ 203.863946] LDISKFS-fs (dm-0): recovery complete [ 203.871363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.895324] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 204.031950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.072658] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 205.953474] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 209.380509] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 209.405913] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 212.227746] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 212.985197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 218.274648] Lustre: DEBUG MARKER: == replay-single test 0b: ensure object created after recover exists. (3284) ========================================================== 01:24:55 (1784093095) [ 219.586500] Lustre: Failing over lustre-OST0000 [ 219.618308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 219.618938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.636954] Lustre: Skipped 1 previous similar message [ 219.641225] LustreError: 8413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 219.652640] LustreError: 8413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 219.702351] Lustre: server umount lustre-OST0000 complete [ 224.740148] LustreError: 8958:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 224.754668] LustreError: 8958:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 234.097701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.218818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.494068] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 235.805990] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 235.806031] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 235.814747] Lustre: Skipped 3 previous similar messages [ 237.189320] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.225287] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 242.947656] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 248.251106] Lustre: DEBUG MARKER: == replay-single test 0c: check replay-barrier =========== 01:25:25 (1784093125) [ 252.794528] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 253.900085] Lustre: Failing over lustre-MDT0000 [ 254.011179] Lustre: server umount lustre-MDT0000 complete [ 254.946495] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.946495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.946507] Lustre: Skipped 1 previous similar message [ 254.946821] LustreError: 6496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 254.955690] Lustre: Skipped 1 previous similar message [ 254.978743] LustreError: 6496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 271.329380] Lustre: 3629:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784093132/real 1784093132] req@ffff9b3bff87c380 x1870757038779264/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784093148 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 271.336187] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 271.341864] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 271.343893] LDISKFS-fs (dm-0): recovery complete [ 271.350612] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 271.357066] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 271.360345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.545455] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3bfe264e00 x1870757038787840/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 280.670993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 282.984991] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 284.219560] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 284.227283] Lustre: lustre-MDT0000: Denying connection for new client 51631985-c0f5-4b53-8413-8adb87e3032f (at 192.168.201.12@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 286.180673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 286.187091] Lustre: Skipped 1 previous similar message [ 289.555668] Lustre: lustre-MDT0000: Denying connection for new client 51631985-c0f5-4b53-8413-8adb87e3032f (at 192.168.201.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 294.670264] Lustre: lustre-MDT0000: Denying connection for new client 51631985-c0f5-4b53-8413-8adb87e3032f (at 192.168.201.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 299.791296] Lustre: lustre-MDT0000: Denying connection for new client 51631985-c0f5-4b53-8413-8adb87e3032f (at 192.168.201.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 301.544169] hrtimer: interrupt took 3634799 ns [ 304.909505] Lustre: lustre-MDT0000: Denying connection for new client 51631985-c0f5-4b53-8413-8adb87e3032f (at 192.168.201.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:49 [ 315.150142] Lustre: lustre-MDT0000: Denying connection for new client 51631985-c0f5-4b53-8413-8adb87e3032f (at 192.168.201.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:39 [ 315.174499] Lustre: Skipped 1 previous similar message [ 335.629681] Lustre: lustre-MDT0000: Denying connection for new client 51631985-c0f5-4b53-8413-8adb87e3032f (at 192.168.201.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:18 [ 335.639579] Lustre: Skipped 3 previous similar messages [ 354.500348] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 354.505161] Lustre: 18188:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a6b570d0-2eca-4e24-8689-238d22a67d99@ [ 354.521338] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 354.542050] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 354.542754] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 354.553542] Lustre: Skipped 2 previous similar messages [ 354.578657] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:65) [ 354.579151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:65) [ 362.539500] Lustre: DEBUG MARKER: == replay-single test 0d: expired recovery with no clients ========================================================== 01:27:19 (1784093239) [ 367.784373] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 369.134091] Lustre: Failing over lustre-MDT0000 [ 369.248409] Lustre: server umount lustre-MDT0000 complete [ 370.147066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 370.150726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 370.155403] Lustre: Skipped 1 previous similar message [ 370.157245] LustreError: 8343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.163295] LustreError: 8343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 386.210558] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 386.212738] LDISKFS-fs (dm-0): recovery complete [ 386.219394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.298227] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 386.463150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.105774] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 390.371689] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 390.378120] Lustre: lustre-MDT0000: Denying connection for new client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 390.387761] Lustre: Skipped 3 previous similar messages [ 391.651704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 456.976359] Lustre: lustre-MDT0000: Denying connection for new client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:03 [ 456.984101] Lustre: Skipped 12 previous similar messages [ 460.500355] Lustre: lustre-MDT0000: recovery is timed out, evict stale exports [ 460.507759] Lustre: 19914:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 51631985-c0f5-4b53-8413-8adb87e3032f@ [ 460.521853] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 460.532105] Lustre: lustre-MDT0000: Recovery over after 1:10, of 2 clients 1 recovered and 1 was evicted. [ 460.536021] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 460.545427] Lustre: Skipped 2 previous similar messages [ 460.559308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:97) [ 460.559865] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:97) [ 467.368278] Lustre: DEBUG MARKER: == replay-single test 1: simple create =================== 01:29:04 (1784093344) [ 471.275369] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 472.336050] Lustre: Failing over lustre-MDT0000 [ 472.454706] Lustre: server umount lustre-MDT0000 complete [ 473.569113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.573175] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.581976] Lustre: Skipped 5 previous similar messages [ 473.591258] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 489.005485] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 489.007332] LDISKFS-fs (dm-0): recovery complete [ 489.011927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.088626] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.231545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.980761] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 492.813645] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 494.562736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 494.605891] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 494.637265] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:129) [ 494.639433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:44 to 0x280000401:129) [ 497.892865] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 498.867218] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 504.144400] Lustre: DEBUG MARKER: == replay-single test 2a: touch ========================== 01:29:40 (1784093380) [ 508.461894] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 509.506978] Lustre: Failing over lustre-MDT0000 [ 509.699716] Lustre: server umount lustre-MDT0000 complete [ 509.922138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.922153] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.941879] Lustre: Skipped 3 previous similar messages [ 526.297817] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 526.300578] LDISKFS-fs (dm-0): recovery complete [ 526.310415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.368597] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 528.659607] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 528.987839] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 531.948432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 531.953404] Lustre: Skipped 3 previous similar messages [ 532.010700] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 532.041063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 532.041281] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:161) [ 535.317560] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 536.234888] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 541.212703] Lustre: DEBUG MARKER: == replay-single test 2b: touch ========================== 01:30:17 (1784093417) [ 545.503481] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 546.491358] Lustre: Failing over lustre-MDT0000 [ 546.654097] Lustre: server umount lustre-MDT0000 complete [ 547.297844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.304644] Lustre: Skipped 3 previous similar messages [ 547.307723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 562.657026] Lustre: 3631:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784093424/real 1784093424] req@ffff9b3bfe265c00 x1870757038932480/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784093440 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 562.673871] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.032142] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 563.033575] LDISKFS-fs (dm-0): recovery complete [ 563.038380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 572.897506] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3becd5bb80 x1870757038941184/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 574.238617] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 574.899855] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 578.584380] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 578.611079] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:44 to 0x2c0000401:193) [ 578.611147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 581.085632] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 581.986357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 586.442395] Lustre: DEBUG MARKER: == replay-single test 2c: setstripe replay =============== 01:31:03 (1784093463) [ 590.274548] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 591.352725] Lustre: Failing over lustre-MDT0000 [ 591.501722] Lustre: server umount lustre-MDT0000 complete [ 593.889300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 604.130477] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.139034] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 72 previous similar messages [ 607.675755] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 607.677679] LDISKFS-fs (dm-0): recovery complete [ 607.683765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 607.760738] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 607.883467] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 609.960272] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 610.579055] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 613.347219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 613.354203] Lustre: Skipped 7 previous similar messages [ 613.411799] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 613.432884] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:225) [ 613.433072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 616.837282] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 617.680781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 622.628272] Lustre: DEBUG MARKER: == replay-single test 2d: setdirstripe replay ============ 01:31:39 (1784093499) [ 626.422731] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 627.481975] Lustre: Failing over lustre-MDT0000 [ 627.602904] Lustre: server umount lustre-MDT0000 complete [ 628.704973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.705158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 628.711938] Lustre: Skipped 7 previous similar messages [ 643.886212] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 643.887890] LDISKFS-fs (dm-0): recovery complete [ 643.896200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.976375] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 644.103171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 644.106844] Lustre: Skipped 3 previous similar messages [ 644.130177] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 645.859660] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 649.227842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 649.228234] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:257) [ 651.705267] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 652.553444] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 657.542429] Lustre: DEBUG MARKER: == replay-single test 2e: O_CREAT|O_EXCL create replay === 01:32:14 (1784093534) [ 658.138254] Lustre: *** cfs_fail_loc=13b, val=315*** [ 658.141231] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 658.146037] LustreError: 6500:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bfc951500 x1870757028295552/t38654705666(0) o35->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:611/0 lens 392/456 e 0 to 0 dl 1784093546 ref 1 fl Interpret:/600/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 663.761815] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 664.794510] Lustre: Failing over lustre-MDT0000 [ 664.989250] Lustre: server umount lustre-MDT0000 complete [ 669.665126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 681.805585] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 681.807771] LDISKFS-fs (dm-0): recovery complete [ 681.816290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.982754] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 682.253560] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 682.258018] Lustre: Skipped 1 previous similar message [ 684.118246] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 687.111992] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 687.118419] Lustre: Skipped 1 previous similar message [ 687.132746] Lustre: 6500:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3ac9763480 x1870757028295552/t38654705666(0) o35->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:640/0 lens 392/456 e 0 to 0 dl 1784093575 ref 1 fl Interpret:/602/0 rc 0/0 job:'openfile.0' uid:0 gid:0 projid:0 [ 687.151876] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:289) [ 687.153352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 690.188378] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 691.088315] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 695.957066] Lustre: DEBUG MARKER: == replay-single test 3a: replay failed open(O_DIRECTORY) ========================================================== 01:32:52 (1784093572) [ 699.730663] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 700.713168] Lustre: Failing over lustre-MDT0000 [ 700.853813] Lustre: server umount lustre-MDT0000 complete [ 716.851046] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 716.852610] LDISKFS-fs (dm-0): recovery complete [ 716.856571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.938299] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 716.941692] LustreError: Skipped 1 previous similar message [ 717.083415] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 719.218992] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 722.452052] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:321) [ 722.452070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 725.277479] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 726.071104] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 730.751954] Lustre: DEBUG MARKER: == replay-single test 3b: replay failed open -ENOMEM ===== 01:33:27 (1784093607) [ 734.307103] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 734.793980] Lustre: *** cfs_fail_loc=114, val=0*** [ 736.215306] Lustre: Failing over lustre-MDT0000 [ 736.379905] Lustre: server umount lustre-MDT0000 complete [ 737.760970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 737.764261] LustreError: Skipped 1 previous similar message [ 752.896205] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 752.898283] LDISKFS-fs (dm-0): recovery complete [ 752.902894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 753.118187] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 755.222470] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 758.247601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 758.250682] Lustre: Skipped 15 previous similar messages [ 758.280401] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:353) [ 758.280505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 761.046542] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 761.889190] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 767.094472] Lustre: DEBUG MARKER: == replay-single test 3c: replay failed open -ENOMEM ===== 01:34:03 (1784093643) [ 771.448210] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 771.956717] Lustre: *** cfs_fail_loc=128, val=0*** [ 773.405958] Lustre: Failing over lustre-MDT0000 [ 773.539327] Lustre: server umount lustre-MDT0000 complete [ 773.605090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.611059] Lustre: Skipped 14 previous similar messages [ 790.001395] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 790.002984] LDISKFS-fs (dm-0): recovery complete [ 790.011251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.197544] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 792.127130] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 795.668589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 795.669135] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:195 to 0x2c0000401:385) [ 798.498365] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 799.301991] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 803.684126] Lustre: DEBUG MARKER: == replay-single test 4a: |x| 10 open(O_CREAT)s ========== 01:34:40 (1784093680) [ 807.425658] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 808.574417] Lustre: Failing over lustre-MDT0000 [ 808.718074] Lustre: server umount lustre-MDT0000 complete [ 824.044566] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 824.046377] LDISKFS-fs (dm-0): recovery complete [ 824.050347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.230056] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 826.072932] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 826.637521] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 826.643547] Lustre: Skipped 3 previous similar messages [ 829.499404] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 829.502283] Lustre: Skipped 3 previous similar messages [ 829.518916] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:391 to 0x2c0000401:417) [ 829.519136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:391 to 0x280000401:417) [ 832.082947] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 832.939459] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 837.784298] Lustre: DEBUG MARKER: == replay-single test 4b: |x| rm 10 files ================ 01:35:14 (1784093714) [ 841.612637] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 842.569877] Lustre: Failing over lustre-MDT0000 [ 842.719088] Lustre: server umount lustre-MDT0000 complete [ 858.031939] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 858.033501] LDISKFS-fs (dm-0): recovery complete [ 858.039369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.102200] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 858.106023] LustreError: Skipped 3 previous similar messages [ 858.238068] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 860.151489] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 863.783753] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:423 to 0x2c0000401:449) [ 863.783753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:423 to 0x280000401:449) [ 866.101451] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 866.745340] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 871.059750] Lustre: DEBUG MARKER: == replay-single test 5: |x| 220 open(O_CREAT) =========== 01:35:47 (1784093747) [ 874.592616] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 878.005210] Lustre: Failing over lustre-MDT0000 [ 878.226236] Lustre: server umount lustre-MDT0000 complete [ 879.076288] LustreError: 6498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.081219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 879.085124] LustreError: 6498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 113 previous similar messages [ 879.098527] LustreError: Skipped 3 previous similar messages [ 893.136213] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 893.138330] LDISKFS-fs (dm-0): recovery complete [ 893.142162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.788068] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 899.867675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:577) [ 899.868126] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:577) [ 902.526941] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 903.433810] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 915.143843] Lustre: DEBUG MARKER: == replay-single test 6a: mkdir + contained create ======= 01:36:32 (1784093792) [ 917.937313] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 918.632053] Lustre: Failing over lustre-MDT0000 [ 918.797591] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 918.832636] Lustre: server umount lustre-MDT0000 complete [ 933.624642] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 933.626711] LDISKFS-fs (dm-0): recovery complete [ 933.629539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.737512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.740791] Lustre: Skipped 7 previous similar messages [ 933.752522] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 933.756229] Lustre: Skipped 1 previous similar message [ 935.326243] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 939.008139] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:609) [ 939.008171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:609) [ 941.314819] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 941.947631] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 947.772889] Lustre: DEBUG MARKER: == replay-single test 6b: |X| rmdir ====================== 01:37:04 (1784093824) [ 950.639587] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 951.450761] Lustre: Failing over lustre-MDT0000 [ 951.649817] Lustre: server umount lustre-MDT0000 complete [ 966.858201] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 966.860051] LDISKFS-fs (dm-0): recovery complete [ 966.866703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.668816] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 972.285176] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:641) [ 972.285566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:641) [ 974.202891] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 974.757893] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 978.619263] Lustre: DEBUG MARKER: == replay-single test 7: mkdir |X| contained create ====== 01:37:35 (1784093855) [ 981.722267] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 982.551083] Lustre: Failing over lustre-MDT0000 [ 982.741880] Lustre: server umount lustre-MDT0000 complete [ 997.720834] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 997.722235] LDISKFS-fs (dm-0): recovery complete [ 997.728534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.403504] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1003.008754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:673) [ 1003.009256] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:673) [ 1004.999266] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1005.588709] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1009.321329] Lustre: DEBUG MARKER: == replay-single test 8: creat open |X| close ============ 01:38:06 (1784093886) [ 1012.001817] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1012.765199] Lustre: Failing over lustre-MDT0000 [ 1012.922962] Lustre: server umount lustre-MDT0000 complete [ 1027.057911] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1027.059617] LDISKFS-fs (dm-0): recovery complete [ 1027.062975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.788891] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1032.674189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1032.676832] Lustre: Skipped 31 previous similar messages [ 1032.728676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:705) [ 1032.728676] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:705) [ 1034.843463] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1035.455791] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1039.159818] Lustre: DEBUG MARKER: == replay-single test 9: |X| create (same inum/gen) ====== 01:38:36 (1784093916) [ 1042.047871] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1042.740096] Lustre: Failing over lustre-MDT0000 [ 1042.912638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.918312] Lustre: Skipped 28 previous similar messages [ 1042.920306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1042.922282] Lustre: Skipped 3 previous similar messages [ 1042.978696] Lustre: server umount lustre-MDT0000 complete [ 1057.340358] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1057.341644] LDISKFS-fs (dm-0): recovery complete [ 1057.344773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1059.057318] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1062.921638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:737) [ 1062.921688] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:737) [ 1065.039392] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1065.617799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1069.244668] Lustre: DEBUG MARKER: == replay-single test 10: create |X| rename unlink ======= 01:39:06 (1784093946) [ 1071.938780] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1072.693959] Lustre: Failing over lustre-MDT0000 [ 1072.806568] Lustre: server umount lustre-MDT0000 complete [ 1087.479290] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1087.481234] LDISKFS-fs (dm-0): recovery complete [ 1087.486944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1087.647536] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1087.650681] Lustre: Skipped 4 previous similar messages [ 1087.757135] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1087.761309] Lustre: Skipped 7 previous similar messages [ 1089.248645] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1093.108191] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [ 1093.112314] Lustre: Skipped 7 previous similar messages [ 1093.126890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:560 to 0x2c0000401:769) [ 1093.126908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:560 to 0x280000401:769) [ 1095.076935] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1095.611542] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1099.388493] Lustre: DEBUG MARKER: == replay-single test 11: create open write rename |X| create-old-name read ========================================================== 01:39:36 (1784093976) [ 1102.329040] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1103.102461] Lustre: Failing over lustre-MDT0000 [ 1103.276026] Lustre: server umount lustre-MDT0000 complete [ 1117.556544] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1117.558046] LDISKFS-fs (dm-0): recovery complete [ 1117.560862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1117.600477] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.605193] LustreError: Skipped 7 previous similar messages [ 1119.207047] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1122.833525] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:801) [ 1122.833564] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:801) [ 1124.905480] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1125.509790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1129.109452] Lustre: DEBUG MARKER: == replay-single test 12: open, unlink |X| close ========= 01:40:06 (1784094006) [ 1131.923819] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1132.622122] Lustre: Failing over lustre-MDT0000 [ 1132.775086] Lustre: server umount lustre-MDT0000 complete [ 1146.896713] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1146.898023] LDISKFS-fs (dm-0): recovery complete [ 1146.901668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1148.455107] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1152.509168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:833) [ 1152.509167] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:771 to 0x2c0000401:833) [ 1154.353612] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1154.904396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1158.507969] Lustre: DEBUG MARKER: == replay-single test 13: open chmod 0 |x| write close === 01:40:35 (1784094035) [ 1161.239554] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1162.007292] Lustre: Failing over lustre-MDT0000 [ 1162.146673] Lustre: server umount lustre-MDT0000 complete [ 1162.722243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1162.725499] LustreError: Skipped 6 previous similar messages [ 1176.487851] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1176.490050] LDISKFS-fs (dm-0): recovery complete [ 1176.494193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.969342] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1181.688313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:865) [ 1181.688926] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:835 to 0x2c0000401:865) [ 1183.720706] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1184.309396] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1187.828879] Lustre: DEBUG MARKER: == replay-single test 14: open(O_CREAT), unlink |X| close ========================================================== 01:41:04 (1784094064) [ 1190.577555] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1191.320883] Lustre: Failing over lustre-MDT0000 [ 1191.505775] Lustre: server umount lustre-MDT0000 complete [ 1205.771636] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1205.773784] LDISKFS-fs (dm-0): recovery complete [ 1205.777125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1207.426597] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1211.389877] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:867 to 0x2c0000401:897) [ 1211.390070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:771 to 0x280000401:897) [ 1213.413885] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1213.996132] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1218.024500] Lustre: DEBUG MARKER: == replay-single test 15: open(O_CREAT), unlink |X| touch new, close ========================================================== 01:41:35 (1784094095) [ 1221.656308] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1222.866670] Lustre: Failing over lustre-MDT0000 [ 1222.991261] Lustre: server umount lustre-MDT0000 complete [ 1242.561195] Lustre: 3629:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784094104/real 1784094104] req@ffff9b3bf586d880 x1870757039489792/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784094120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1242.978680] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1242.981248] LDISKFS-fs (dm-0): recovery complete [ 1242.989172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1252.834268] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3bf8722680 x1870757039497344/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1255.609898] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1258.570797] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:929) [ 1258.570905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:899 to 0x280000401:929) [ 1263.271355] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1264.388688] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1271.821566] Lustre: DEBUG MARKER: == replay-single test 16: |X| open(O_CREAT), unlink, touch new, unlink new ========================================================== 01:42:28 (1784094148) [ 1279.105681] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1281.025679] Lustre: Failing over lustre-MDT0000 [ 1281.506477] Lustre: server umount lustre-MDT0000 complete [ 1299.360405] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1299.361982] LDISKFS-fs (dm-0): recovery complete [ 1299.366572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.884572] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1304.619681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:961) [ 1304.621106] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:899 to 0x2c0000401:961) [ 1307.708728] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1308.767872] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1314.280622] Lustre: DEBUG MARKER: == replay-single test 17: |X| open(O_CREAT), |replay| close ========================================================== 01:43:10 (1784094190) [ 1318.682830] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1319.757341] Lustre: Failing over lustre-MDT0000 [ 1319.884968] Lustre: server umount lustre-MDT0000 complete [ 1337.640119] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1337.643242] LDISKFS-fs (dm-0): recovery complete [ 1337.650769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1340.851893] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1343.003847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:931 to 0x280000401:993) [ 1343.003847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:963 to 0x2c0000401:993) [ 1346.991491] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1348.061772] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1354.099068] Lustre: DEBUG MARKER: == replay-single test 18: open(O_CREAT), unlink, touch new, close, touch, unlink ========================================================== 01:43:50 (1784094230) [ 1359.298119] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1360.732049] Lustre: Failing over lustre-MDT0000 [ 1360.967769] Lustre: server umount lustre-MDT0000 complete [ 1378.936441] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1378.938817] LDISKFS-fs (dm-0): recovery complete [ 1378.945130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.168564] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1379.174239] Lustre: Skipped 7 previous similar messages [ 1381.870483] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1384.501347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:995 to 0x280000401:1025) [ 1384.501472] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:995 to 0x2c0000401:1025) [ 1387.795938] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1388.698755] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1394.182399] Lustre: DEBUG MARKER: == replay-single test 19: mcreate, open, write, rename === 01:44:30 (1784094270) [ 1399.229659] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1400.712884] Lustre: Failing over lustre-MDT0000 [ 1400.855060] Lustre: server umount lustre-MDT0000 complete [ 1403.157863] LustreError: 6498:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.180290] LustreError: 6498:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 242 previous similar messages [ 1418.992197] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1418.994343] LDISKFS-fs (dm-0): recovery complete [ 1419.000616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.684774] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1424.404336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1027 to 0x280000401:1057) [ 1424.404515] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1057) [ 1427.934806] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1428.971360] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1434.714977] Lustre: DEBUG MARKER: == replay-single test 20a: |X| open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 01:45:11 (1784094311) [ 1439.955654] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1441.391311] Lustre: Failing over lustre-MDT0000 [ 1441.676826] Lustre: server umount lustre-MDT0000 complete [ 1458.741612] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1458.744346] LDISKFS-fs (dm-0): recovery complete [ 1458.751144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.942699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.950221] Lustre: Skipped 14 previous similar messages [ 1461.195369] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1464.371818] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1089) [ 1464.372039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1059 to 0x280000401:1089) [ 1467.559600] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1468.689571] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1475.355778] Lustre: DEBUG MARKER: == replay-single test 20b: write, unlink, eviction, replay (test mds_cleanup_orphans) ========================================================== 01:45:51 (1784094351) [ 1478.949482] Lustre: 75109:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 9d96a3b9-8c31-46b4-b224-2e30ff5b404a at adminstrative request [ 1482.497658] Lustre: Failing over lustre-MDT0000 [ 1482.646661] Lustre: server umount lustre-MDT0000 complete [ 1498.518062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.446064] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1503.761320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1091 to 0x280000401:1121) [ 1503.765345] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1027 to 0x2c0000401:1121) [ 1508.969164] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1510.135062] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1514.392826] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 1521.370187] Lustre: DEBUG MARKER: before 3156, after 3156 [ 1526.414813] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content ========================================================== 01:46:42 (1784094402) [ 1527.130502] Lustre: 77132:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 9d96a3b9-8c31-46b4-b224-2e30ff5b404a at adminstrative request [ 1534.138583] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 01:46:50 (1784094410) [ 1538.664229] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1539.808754] Lustre: Failing over lustre-MDT0000 [ 1539.945251] Lustre: server umount lustre-MDT0000 complete [ 1561.056221] Lustre: 3629:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784094422/real 1784094422] req@ffff9b3bffa4f800 x1870757039695104/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784094438 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1562.165782] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1562.167503] LDISKFS-fs (dm-0): recovery complete [ 1562.200193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.299985] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x83d137b8a1463e12 [ 1571.307545] Lustre: MGC192.168.201.112@tcp: Connection restored to 0@lo (at 0@lo) [ 1571.314970] Lustre: Skipped 55 previous similar messages [ 1577.198604] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1123 to 0x2c0000401:1153) [ 1577.204626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1124 to 0x280000401:1153) [ 1577.444608] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1589.790527] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1591.754657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1602.333227] Lustre: DEBUG MARKER: == replay-single test 22: open(O_CREAT), |X| unlink, replay, close (test mds_cleanup_orphans) ========================================================== 01:47:57 (1784094477) [ 1612.941461] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1615.439291] Lustre: Failing over lustre-MDT0000 [ 1616.098458] Lustre: server umount lustre-MDT0000 complete [ 1617.891384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.931884] Lustre: Skipped 57 previous similar messages [ 1634.280658] Lustre: 3628:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784094495/real 1784094495] req@ffff9b3ac99a8700 x1870757039736320/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784094511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1634.319600] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1634.339370] LustreError: Skipped 11 previous similar messages [ 1642.059742] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1642.067175] LDISKFS-fs (dm-0): recovery complete [ 1642.077948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.256145] LustreError: 80381:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1644.264992] LustreError: 80381:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff9b3becf3e680 x1870757039742336/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1784094521 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1644.297988] LustreError: 80381:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1645.931443] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1645.943564] Lustre: Skipped 12 previous similar messages [ 1649.551567] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1650.222727] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 1650.234656] Lustre: Skipped 12 previous similar messages [ 1650.273847] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1155 to 0x2c0000401:1185) [ 1650.278156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1155 to 0x280000401:1185) [ 1658.080099] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1659.708996] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1667.311242] Lustre: DEBUG MARKER: == replay-single test 23: open(O_CREAT), |X| unlink touch new, replay, close (test mds_cleanup_orphans) ========================================================== 01:49:03 (1784094543) [ 1674.387770] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1676.177829] Lustre: Failing over lustre-MDT0000 [ 1676.635890] Lustre: server umount lustre-MDT0000 complete [ 1680.865758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1680.881674] LustreError: Skipped 8 previous similar messages [ 1697.249305] Lustre: 3630:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784094558/real 1784094558] req@ffff9b3bfce11f80 x1870757039772416/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784094574 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1698.406796] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1698.411275] LDISKFS-fs (dm-0): recovery complete [ 1698.419403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1707.488634] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3bed291500 x1870757039782272/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1711.580951] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1713.240441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1187 to 0x280000401:1217) [ 1713.250561] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1187 to 0x2c0000401:1217) [ 1720.053718] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1721.430784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1730.187554] Lustre: DEBUG MARKER: == replay-single test 24: open(O_CREAT), replay, unlink, close (test mds_cleanup_orphans) ========================================================== 01:50:05 (1784094605) [ 1737.635848] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1739.643895] Lustre: Failing over lustre-MDT0000 [ 1739.851404] Lustre: server umount lustre-MDT0000 complete [ 1760.225241] Lustre: 3629:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784094621/real 1784094621] req@ffff9b3bfe094a80 x1870757039809024/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784094637 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1761.917176] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1761.923228] LDISKFS-fs (dm-0): recovery complete [ 1761.945527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1774.685811] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1776.155462] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1249) [ 1776.157057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1219 to 0x280000401:1249) [ 1783.259309] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1784.785921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1792.351426] Lustre: DEBUG MARKER: == replay-single test 25: open(O_CREAT), unlink, replay, close (test mds_cleanup_orphans) ========================================================== 01:51:08 (1784094668) [ 1799.775184] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1801.832619] Lustre: Failing over lustre-MDT0000 [ 1802.075164] Lustre: server umount lustre-MDT0000 complete [ 1823.206657] Lustre: 3630:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784094684/real 1784094684] req@ffff9b3bc6c67480 x1870757039845120/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784094700 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1823.285717] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1823.291948] LDISKFS-fs (dm-0): recovery complete [ 1823.306956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.442349] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3bfc974000 x1870757039853440/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1837.929142] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1839.166795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1219 to 0x2c0000401:1281) [ 1839.168157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1251 to 0x280000401:1281) [ 1846.420670] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1848.037404] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1856.221948] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 01:52:12 (1784094732) [ 1862.551739] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1864.396545] Lustre: Failing over lustre-MDT0000 [ 1864.743482] Lustre: server umount lustre-MDT0000 complete [ 1886.176215] Lustre: 3629:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784094747/real 1784094747] req@ffff9b3bf85a1c00 x1870757039880576/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784094763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1886.646369] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1886.648346] LDISKFS-fs (dm-0): recovery complete [ 1886.658960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.428652] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x83d137b8a14659a9 [ 1896.758484] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 1896.774778] Lustre: Skipped 8 previous similar messages [ 1901.463196] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1902.248932] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1283 to 0x2c0000401:1313) [ 1902.249515] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1283 to 0x280000401:1313) [ 1910.091854] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1911.406936] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1919.508514] Lustre: DEBUG MARKER: == replay-single test 27: |X| open(O_CREAT), unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 01:53:15 (1784094795) [ 1926.771242] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1928.604982] Lustre: Failing over lustre-MDT0000 [ 1928.854420] Lustre: server umount lustre-MDT0000 complete [ 1948.624148] Lustre: 3630:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784094810/real 1784094810] req@ffff9b3bf5cbc380 x1870757039916032/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784094826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1950.577210] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 1950.580342] LDISKFS-fs (dm-0): recovery complete [ 1950.585737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.880614] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3bf8262300 x1870757039924352/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 1964.021837] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1964.667342] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1315 to 0x2c0000401:1345) [ 1964.671779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1315 to 0x280000401:1345) [ 1974.370801] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1976.045462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1984.979334] Lustre: DEBUG MARKER: == replay-single test 28: open(O_CREAT), |X| unlink two, close one, replay, close one (test mds_cleanup_orphans) ========================================================== 01:54:21 (1784094861) [ 1992.706555] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1994.665858] Lustre: Failing over lustre-MDT0000 [ 1994.983917] Lustre: server umount lustre-MDT0000 complete [ 2005.475647] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2005.490027] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 300 previous similar messages [ 2017.495098] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2017.496915] LDISKFS-fs (dm-0): recovery complete [ 2017.502256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.849597] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x83d137b8a14665aa [ 2026.205521] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2026.621572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1347 to 0x280000401:1377) [ 2026.623335] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1347 to 0x2c0000401:1377) [ 2035.840440] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2037.666722] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2046.074925] Lustre: DEBUG MARKER: == replay-single test 29: open(O_CREAT), |X| unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 01:55:22 (1784094922) [ 2053.381343] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2055.378783] Lustre: Failing over lustre-MDT0000 [ 2055.696695] Lustre: server umount lustre-MDT0000 complete [ 2072.548101] Lustre: 3629:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784094934/real 1784094934] req@ffff9b3bffa4d180 x1870757039988992/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784094950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2072.599126] Lustre: 3629:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2077.963166] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2077.965678] LDISKFS-fs (dm-0): recovery complete [ 2077.971227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.788351] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x83d137b8a1466b5a [ 2083.024123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.030893] Lustre: Skipped 9 previous similar messages [ 2087.348461] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2088.486451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1379 to 0x280000401:1409) [ 2088.496232] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1379 to 0x2c0000401:1409) [ 2097.144769] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2099.068001] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2108.350310] Lustre: DEBUG MARKER: == replay-single test 30: open(O_CREAT) two, unlink two, replay, close two (test mds_cleanup_orphans) ========================================================== 01:56:24 (1784094984) [ 2116.415293] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2118.835879] Lustre: Failing over lustre-MDT0000 [ 2119.241033] Lustre: server umount lustre-MDT0000 complete [ 2140.906361] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2140.909258] LDISKFS-fs (dm-0): recovery complete [ 2140.924508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.249864] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3bffa49f80 x1870757040033408/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2149.961240] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2150.963867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1411 to 0x2c0000401:1441) [ 2150.976259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1411 to 0x280000401:1441) [ 2160.168613] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2162.019784] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2170.699357] Lustre: DEBUG MARKER: == replay-single test 31: open(O_CREAT) two, unlink one, |X| unlink one, close two (test mds_cleanup_orphans) ========================================================== 01:57:26 (1784095046) [ 2177.732812] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2179.840466] Lustre: Failing over lustre-MDT0000 [ 2179.988316] Lustre: server umount lustre-MDT0000 complete [ 2200.293781] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2200.295798] LDISKFS-fs (dm-0): recovery complete [ 2200.301035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2208.232089] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x83d137b8a1467746 [ 2208.244300] Lustre: MGC192.168.201.112@tcp: Connection restored to 0@lo (at 0@lo) [ 2208.258607] Lustre: Skipped 43 previous similar messages [ 2212.460787] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2213.935581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1443 to 0x280000401:1473) [ 2213.936155] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1443 to 0x2c0000401:1473) [ 2221.602390] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2223.179841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2231.242044] Lustre: DEBUG MARKER: == replay-single test 32: close() notices client eviction; close() after client eviction ========================================================== 01:58:27 (1784095107) [ 2232.601070] Lustre: 98715:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 9d96a3b9-8c31-46b4-b224-2e30ff5b404a at adminstrative request [ 2241.722757] Lustre: DEBUG MARKER: == replay-single test 33a: fid seq shouldn't be reused after abort recovery ========================================================== 01:58:38 (1784095118) [ 2247.758671] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2249.316447] Lustre: Failing over lustre-MDT0000 [ 2249.609147] Lustre: server umount lustre-MDT0000 complete [ 2249.702888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2249.725339] Lustre: Skipped 39 previous similar messages [ 2259.963406] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2259.966064] LDISKFS-fs (dm-0): recovery complete [ 2259.973820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2260.063310] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.078229] LustreError: Skipped 9 previous similar messages [ 2260.361510] Lustre: lustre-MDT0000: Aborting client recovery [ 2260.363372] LustreError: 99997:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2260.367733] Lustre: 100032:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2260.370571] LustreError: 100030:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0000-osd: get update log duration 0, retries 0, failed: rc = -108 [ 2260.377791] Lustre: 100032:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a@ [ 2260.383719] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2260.404267] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 2260.426865] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 2260.499741] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1479 to 0x2c0000401:1505) [ 2260.502599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1480 to 0x280000401:1505) [ 2264.443228] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2265.575199] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2283.443471] Lustre: DEBUG MARKER: == replay-single test 33b: test fid seq allocation ======= 01:59:20 (1784095160) [ 2289.673631] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2291.385514] Lustre: Failing over lustre-MDT0000 [ 2291.598893] Lustre: server umount lustre-MDT0000 complete [ 2301.947136] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2301.949488] LDISKFS-fs (dm-0): recovery complete [ 2301.961159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.205737] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2302.238713] Lustre: lustre-MDT0000: Aborting client recovery [ 2302.242679] LustreError: 101861:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2302.248537] Lustre: 101894:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2302.255353] Lustre: 101894:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2302.260504] Lustre: 101894:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2302.266382] Lustre: 101894:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2302.271775] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2302.278936] Lustre: lustre-MDT0000-osd: cancel update llog [0x200015bc0:0x1:0x0] [ 2302.289895] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000403:0x1:0x0] [ 2302.348770] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1511 to 0x2c0000401:1537) [ 2302.352306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1511 to 0x280000401:1537) [ 2305.722064] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2307.557662] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2318.289794] Lustre: *** cfs_fail_loc=1311, val=0*** [ 2318.295635] Lustre: Skipped 1 previous similar message [ 2323.455426] Lustre: DEBUG MARKER: == replay-single test 34: abort recovery before client does replay (test mds_cleanup_orphans) ========================================================== 02:00:00 (1784095200) [ 2329.128431] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2330.540056] Lustre: Failing over lustre-MDT0000 [ 2330.692930] Lustre: server umount lustre-MDT0000 complete [ 2333.153111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2333.156961] LustreError: Skipped 7 previous similar messages [ 2340.744709] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2340.747189] LDISKFS-fs (dm-0): recovery complete [ 2340.760255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2341.050664] Lustre: lustre-MDT0000: Aborting client recovery [ 2341.053153] LustreError: 103729:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2341.060412] Lustre: 103762:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2341.063362] Lustre: 103762:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2341.067348] Lustre: 103762:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2341.081901] Lustre: 103762:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2341.090333] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2341.096976] Lustre: lustre-MDT0000-osd: cancel update llog [0x200016778:0x1:0x0] [ 2341.107531] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e8:0x1:0x0] [ 2341.143872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1569) [ 2341.144528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1543 to 0x280000401:1569) [ 2344.605600] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2346.469535] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2363.368277] Lustre: DEBUG MARKER: == replay-single test 35: test recovery from llog for unlink op ========================================================== 02:00:39 (1784095239) [ 2364.193579] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 2364.197933] LustreError: 6496:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3becd7ad80 x1870757029298688/t201863462916(0) o36->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:52/0 lens 512/456 e 0 to 0 dl 1784095252 ref 1 fl Interpret:/200/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 [ 2367.799760] Lustre: Failing over lustre-MDT0000 [ 2369.984779] Lustre: server umount lustre-MDT0000 complete [ 2376.729507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.064542] Lustre: lustre-MDT0000: Aborting client recovery [ 2377.067486] LustreError: 105052:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2377.072741] Lustre: 105085:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2377.080637] Lustre: 105085:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2377.088602] Lustre: 105085:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2377.093644] Lustre: 105085:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2377.098537] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2377.106745] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017330:0x1:0x0] [ 2377.122275] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007e9:0x1:0x0] [ 2377.168031] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1601) [ 2377.177173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1601) [ 2380.639459] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2382.322633] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2398.858982] Lustre: DEBUG MARKER: SKIP: replay-single test_36 skipping ALWAYS excluded test 36 [ 2400.206873] Lustre: DEBUG MARKER: == replay-single test 37: abort recovery before client does replay (test mds_cleanup_orphans for directories) ========================================================== 02:01:16 (1784095276) [ 2406.122558] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2408.091894] Lustre: Failing over lustre-MDT0000 [ 2408.229710] Lustre: server umount lustre-MDT0000 complete [ 2417.500623] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2417.506122] LDISKFS-fs (dm-0): recovery complete [ 2417.516504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.808773] Lustre: lustre-MDT0000: Aborting client recovery [ 2417.811573] LustreError: 107028:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 2417.816466] Lustre: 107063:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 2417.820617] Lustre: 107063:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 2417.825525] Lustre: 107063:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 2417.830754] Lustre: 107063:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 2417.835043] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 2417.840838] Lustre: lustre-MDT0000-osd: cancel update llog [0x200017b00:0x1:0x0] [ 2417.851867] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007ea:0x1:0x0] [ 2417.896121] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:1544 to 0x2c0000401:1633) [ 2417.906532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:1571 to 0x280000401:1633) [ 2422.011815] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2423.279041] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 2441.035226] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 02:01:57 (1784095317) [ 2467.157455] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2468.632963] Lustre: Failing over lustre-MDT0000 [ 2468.816427] Lustre: server umount lustre-MDT0000 complete [ 2484.576198] Lustre: 3631:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784095346/real 1784095346] req@ffff9b3bf594fb80 x1870757040319616/t0(0) o400->MGC192.168.201.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784095362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2484.601285] Lustre: 3631:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2487.384586] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2487.388353] LDISKFS-fs (dm-0): recovery complete [ 2487.395051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2494.957555] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x83d137b8a14788af [ 2495.766881] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2495.779433] Lustre: Skipped 9 previous similar messages [ 2498.270675] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2500.641989] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2500.648627] Lustre: Skipped 9 previous similar messages [ 2500.673591] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2034 to 0x2c0000401:2049) [ 2500.673820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2034 to 0x280000401:2049) [ 2505.748232] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2506.787968] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2524.291564] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 02:03:20 (1784095400) [ 2545.168710] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2552.499238] Lustre: Failing over lustre-MDT0000 [ 2552.694359] Lustre: server umount lustre-MDT0000 complete [ 2570.934706] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2570.937315] LDISKFS-fs (dm-0): recovery complete [ 2570.941983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.173132] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2571.178788] Lustre: Skipped 21 previous similar messages [ 2574.150784] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2578.514876] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2450 to 0x2c0000401:2465) [ 2578.514879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2450 to 0x280000401:2465) [ 2581.863834] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2582.821325] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2597.991205] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 02:04:34 (1784095474) [ 2599.471351] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 2600.138361] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2600.141700] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2604.326975] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 02:04:40 (1784095480) [ 2623.515054] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2632.251240] Lustre: Failing over lustre-OST0000 [ 2632.680398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.049143] Lustre: lustre-OST0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 2634.341193] Lustre: server umount lustre-OST0000 complete [ 2637.283263] LustreError: 8413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.296095] LustreError: 8413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 199 previous similar messages [ 2652.787545] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 2652.789702] LDISKFS-fs (dm-2): recovery complete [ 2652.795775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2657.237876] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2710.808502] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 02:06:27 (1784095587) [ 2716.461849] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2718.412389] Lustre: Failing over lustre-MDT0000 [ 2720.628429] Lustre: server umount lustre-MDT0000 complete [ 2738.771296] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 2738.774110] LDISKFS-fs (dm-0): recovery complete [ 2738.781639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.347048] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x83d137b8a14ac0f6 [ 2746.508179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2746.512586] Lustre: Skipped 10 previous similar messages [ 2749.280288] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2752.050314] Lustre: *** cfs_fail_loc=204, val=2147483648*** [ 2752.050770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2913) [ 2755.943266] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2757.015639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2768.352316] Lustre: 116469:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784095629/real 1784095629] req@ffff9b3acb55df80 x1870757040829568/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1784095645 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre-1-0.0' uid:0 gid:0 projid:4294967295 [ 2768.374647] Lustre: 116469:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2768.383863] LustreError: 116469:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 2768.384862] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 2769.441438] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2881) [ 2773.066449] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 02:07:29 (1784095649) [ 2776.530292] LustreError: 6498:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2781.665033] LustreError: 6498:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2781.670148] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnecting [ 2782.809870] LustreError: 6496:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2788.320154] LustreError: 6496:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2788.328922] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnecting [ 2789.892448] LustreError: 11544:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2794.976574] LustreError: 11544:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2794.986458] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnecting [ 2796.234100] LustreError: 52308:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2801.633618] LustreError: 52308:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2802.916945] LustreError: 6496:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2808.288124] LustreError: 6496:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2808.292592] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnecting [ 2808.297539] Lustre: Skipped 1 previous similar message [ 2816.358778] LustreError: 11544:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2816.361277] LustreError: 11544:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2821.600753] LustreError: 11544:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2821.607699] LustreError: 11544:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 1 previous similar message [ 2828.259159] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnecting [ 2828.267943] Lustre: Skipped 2 previous similar messages [ 2836.421296] LustreError: 6497:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_race id 701 sleeping [ 2836.427299] LustreError: 6497:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2841.568441] LustreError: 6497:0:(ldlm_lib.c:1178:target_handle_connect()) cfs_fail_race id 701 awake: rc=0 [ 2841.577678] LustreError: 6497:0:(ldlm_lib.c:1178:target_handle_connect()) Skipped 2 previous similar messages [ 2847.466509] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 02:08:44 (1784095724) [ 2848.957490] LustreError: 6497:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2859.279456] Lustre: lustre-MDT0000: Export ffff9b3bfccf0000 already connecting from 192.168.201.12@tcp [ 2864.402352] Lustre: lustre-MDT0000: Export ffff9b3bfccf0000 already connecting from 192.168.201.12@tcp [ 2869.526420] Lustre: lustre-MDT0000: Export ffff9b3bfccf0000 already connecting from 192.168.201.12@tcp [ 2874.638390] Lustre: lustre-MDT0000: Export ffff9b3bfccf0000 already connecting from 192.168.201.12@tcp [ 2874.646812] Lustre: Skipped 1 previous similar message [ 2879.757884] Lustre: lustre-MDT0000: Export ffff9b3bfccf0000 already connecting from 192.168.201.12@tcp [ 2888.960366] LustreError: 6497:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2888.970606] Lustre: 6497:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9b3bf5918700 x1870757031949184/t0(0) o38->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:0/0 lens 520/416 e 0 to 0 dl 1784095746 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 2889.998123] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnecting [ 2890.011024] Lustre: Skipped 3 previous similar messages [ 2890.016982] LustreError: 11544:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2914.576834] Lustre: lustre-MDT0000: Export ffff9b3bfccf0000 already connecting from 192.168.201.12@tcp [ 2914.580809] Lustre: Skipped 1 previous similar message [ 2930.048132] LustreError: 11544:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2930.056496] Lustre: 11544:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9b3becc5bb80 x1870757031953664/t0(0) o38->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:0/0 lens 520/416 e 0 to 0 dl 1784095787 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2935.060732] LustreError: 6496:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 2959.631902] Lustre: lustre-MDT0000: Export ffff9b3bfccf0000 already connecting from 192.168.201.12@tcp [ 2959.638441] Lustre: Skipped 3 previous similar messages [ 2975.152145] LustreError: 6496:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 2975.159628] Lustre: 6496:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9b3bf5b6bb80 x1870757031957376/t0(0) o38->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:0/0 lens 520/416 e 0 to 0 dl 1784095832 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2979.573779] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnecting [ 2979.584485] Lustre: Skipped 1 previous similar message [ 2979.587443] LustreError: 9475:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3005.199924] Lustre: lustre-MDT0000: Export ffff9b3bfccf0000 already connecting from 192.168.201.12@tcp [ 3005.207149] Lustre: Skipped 3 previous similar messages [ 3019.680148] LustreError: 9475:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3019.687967] Lustre: 9475:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9b3ac7531500 x1870757031960576/t0(0) o38->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:0/0 lens 520/416 e 0 to 0 dl 1784095877 ref 1 fl Complete:H/200/0 rc 0/0 job:'lctl.0' uid:0 gid:0 projid:4294967295 [ 3020.562234] LustreError: 6496:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3060.568145] LustreError: 6496:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 awake [ 3060.572948] Lustre: 6496:0:(service.c:2582:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff9b3bed296a00 x1870757031963520/t0(0) o38->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:0/0 lens 520/416 e 0 to 0 dl 1784095918 ref 1 fl Complete:H/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3061.518777] LustreError: 11544:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout id 704 sleeping for 40000ms [ 3066.227574] LustreError: 11544:0:(ldlm_lib.c:1435:target_handle_connect()) cfs_fail_timeout interrupted [ 3068.977440] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 02:12:25 (1784095945) [ 3074.135700] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3077.341358] Lustre: Failing over lustre-MDT0000 [ 3077.480639] Lustre: server umount lustre-MDT0000 complete [ 3079.649527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3079.658736] LustreError: Skipped 4 previous similar messages [ 3079.663359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.671352] Lustre: Skipped 33 previous similar messages [ 3085.535977] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3085.538319] LDISKFS-fs (dm-0): recovery complete [ 3085.548565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.631200] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3085.636391] LustreError: Skipped 7 previous similar messages [ 3085.760182] Lustre: *** cfs_fail_loc=712, val=0*** [ 3085.767880] LustreError: 43769:0:(service.c:1394:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff9b3bc6c64a80 x1870757040983168/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 [ 3085.841638] Lustre: lustre-MDT0000: Aborting client recovery [ 3085.843035] LustreError: 121317:0:(ldlm_lib.c:3003:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 3085.848585] Lustre: 121351:0:(ldlm_lib.c:2403:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 3085.855425] Lustre: 121351:0:(ldlm_lib.c:2403:target_recovery_overseer()) Skipped 2 previous similar messages [ 3085.859340] Lustre: 121351:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client lustre-MDT0001-mdtlov_UUID@ [ 3085.864980] Lustre: 121351:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 3085.872378] Lustre: lustre-MDT0000: disconnecting 2 stale clients [ 3085.878501] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000182d0:0x1:0x0] [ 3085.887763] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x2400007eb:0x1:0x0] [ 3085.932547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2945) [ 3085.932698] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2913) [ 3088.710944] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3090.920987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3090.924751] Lustre: Skipped 42 previous similar messages [ 3090.931885] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. [ 3101.862740] Lustre: Failing over lustre-MDT0000 [ 3102.058929] Lustre: server umount lustre-MDT0000 complete [ 3117.944852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.895104] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3118.897433] Lustre: Skipped 3 previous similar messages [ 3122.106669] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3123.709458] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3123.713418] Lustre: Skipped 3 previous similar messages [ 3123.741688] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2866 to 0x2c0000401:2945) [ 3123.745298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2867 to 0x280000401:2977) [ 3128.749285] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3129.765187] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3135.476748] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 02:13:32 (1784096012) [ 3135.541673] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnecting [ 3135.545759] Lustre: Skipped 2 previous similar messages [ 3141.707362] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 02:13:38 (1784096018) [ 3142.448261] Lustre: *** cfs_fail_loc=122, val=2147483648*** [ 3142.451207] LustreError: 6506:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bf8261c00 x1870757032022656/t0(0) o700->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:75/0 lens 264/248 e 0 to 0 dl 1784096030 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:4294967295 [ 3161.053615] Lustre: Failing over lustre-MDT0000 [ 3161.245918] Lustre: server umount lustre-MDT0000 complete [ 3176.860471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3177.194457] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3177.196828] Lustre: Skipped 6 previous similar messages [ 3180.367564] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3182.600042] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2947 to 0x2c0000401:2977) [ 3182.600892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2979 to 0x280000401:3009) [ 3186.831868] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3187.921539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3195.136820] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 02:14:31 (1784096071) [ 3196.907245] Lustre: Failing over lustre-OST0000 [ 3197.015928] Lustre: server umount lustre-OST0000 complete [ 3212.593722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3216.652274] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3223.704804] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3224.752580] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3293.322931] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 02:16:09 (1784096169) [ 3298.266275] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3299.709245] Lustre: Failing over lustre-MDT0000 [ 3299.812051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.815465] Lustre: Skipped 3 previous similar messages [ 3299.885199] Lustre: server umount lustre-MDT0000 complete [ 3303.199066] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.211249] LustreError: 6497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 89 previous similar messages [ 3317.573607] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3317.575127] LDISKFS-fs (dm-0): recovery complete [ 3317.580569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3320.416082] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3323.123637] Lustre: *** cfs_fail_loc=216, val=0*** [ 3323.125566] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2998 to 0x2c0000401:3041) [ 3323.126102] LustreError: 127474:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30 [ 3324.192843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3030 to 0x280000401:3073) [ 3389.033115] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 02:17:45 (1784096265) [ 3399.675792] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 02:17:56 (1784096276) [ 3401.561542] Lustre: Failing over lustre-MDT0000 [ 3401.746658] Lustre: server umount lustre-MDT0000 complete [ 3416.299430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.489412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3416.492471] Lustre: Skipped 5 previous similar messages [ 3418.499631] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3421.679569] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 3421.683649] LustreError: 129055:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3becffce00 x1870757032144512/t0(0) o101->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:398/0 lens 328/344 e 0 to 0 dl 1784096353 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 [ 3480.847543] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnected, waiting for 2 clients in recovery for 1:03 [ 3480.860144] Lustre: 129055:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 3480.897382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3105) [ 3480.898258] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3053 to 0x2c0000401:3073) [ 3483.435564] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3484.148159] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3488.720805] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 02:19:25 (1784096365) [ 3490.242578] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3494.204382] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3495.047371] Lustre: Failing over lustre-MDT0000 [ 3495.198690] Lustre: server umount lustre-MDT0000 complete [ 3510.760409] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3510.762014] LDISKFS-fs (dm-0): recovery complete [ 3510.768557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.226909] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3516.429599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3084 to 0x280000401:3137) [ 3516.429640] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3105) [ 3518.854499] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3519.544615] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3523.830193] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 02:20:00 (1784096400) [ 3524.351237] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3528.858718] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3529.662547] Lustre: Failing over lustre-MDT0000 [ 3529.753164] Lustre: server umount lustre-MDT0000 complete [ 3544.640235] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3544.642103] LDISKFS-fs (dm-0): recovery complete [ 3544.646458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.229813] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3550.197718] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3137) [ 3550.197909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3139 to 0x280000401:3169) [ 3552.218473] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3552.813657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3557.000368] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 02:20:33 (1784096433) [ 3557.483749] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3561.758765] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3562.479176] Lustre: Failing over lustre-MDT0000 [ 3562.562155] Lustre: server umount lustre-MDT0000 complete [ 3577.451265] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3577.452684] LDISKFS-fs (dm-0): recovery complete [ 3577.457486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3579.050426] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3582.967058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3171 to 0x280000401:3201) [ 3582.967129] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3169) [ 3588.168207] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 02:21:05 (1784096465) [ 3589.679616] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3589.682041] Lustre: *** cfs_fail_loc=13b, val=2147483648*** [ 3589.684030] LustreError: 6500:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bffa49500 x1870757032195840/t257698037777(0) o35->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:523/0 lens 392/456 e 0 to 0 dl 1784096478 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3590.931426] Lustre: Failing over lustre-MDT0000 [ 3591.114976] Lustre: server umount lustre-MDT0000 complete [ 3604.079752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.745730] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3609.581614] Lustre: 39651:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3bffa0aa00 x1870757032195840/t257698037777(0) o35->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:543/0 lens 392/456 e 0 to 0 dl 1784096498 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3609.587817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3233) [ 3609.589176] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3075 to 0x2c0000401:3201) [ 3611.443459] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3611.972812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3615.592159] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 02:21:32 (1784096492) [ 3615.985789] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3615.988176] LustreError: 6498:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bc8f8b800 x1870757032209152/t261993005072(0) o36->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:593/0 lens 504/448 e 0 to 0 dl 1784096548 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3619.730815] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3620.377493] Lustre: Failing over lustre-MDT0000 [ 3620.464641] Lustre: server umount lustre-MDT0000 complete [ 3634.945314] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3634.946776] LDISKFS-fs (dm-0): recovery complete [ 3634.950727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.509833] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3640.306620] Lustre: 52308:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3acb55e300 x1870757032209152/t261993005072(0) o36->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:617/0 lens 504/2880 e 0 to 0 dl 1784096572 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3640.309175] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3265) [ 3640.309203] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3203 to 0x2c0000401:3233) [ 3642.223057] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3642.818097] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3646.799398] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 02:22:03 (1784096523) [ 3647.250701] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3647.253058] LustreError: 6498:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bf8259c00 x1870757032222464/t266287972368(0) o36->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:624/0 lens 504/448 e 0 to 0 dl 1784096579 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3648.579822] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3651.558546] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3652.333403] Lustre: Failing over lustre-MDT0000 [ 3652.430402] Lustre: server umount lustre-MDT0000 complete [ 3667.303097] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3667.304812] LDISKFS-fs (dm-0): recovery complete [ 3667.308256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3668.954030] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3672.564307] Lustre: 6499:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3becc59180 x1870757032222720/t266287972369(0) o35->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:606/0 lens 392/456 e 0 to 0 dl 1784096561 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3672.567585] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3235 to 0x2c0000401:3265) [ 3672.567670] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3297) [ 3672.577431] Lustre: 6499:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 3677.300870] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 02:22:34 (1784096554) [ 3677.747830] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 3677.749943] Lustre: Skipped 1 previous similar message [ 3677.751564] LustreError: 6497:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3ac803f800 x1870757032235264/t270582939664(0) o36->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:655/0 lens 504/448 e 0 to 0 dl 1784096610 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3677.759103] LustreError: 6497:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 3679.084659] Lustre: *** cfs_fail_loc=115, val=2147483648*** [ 3679.087052] Lustre: Skipped 1 previous similar message [ 3681.992930] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3682.661055] Lustre: Failing over lustre-MDT0000 [ 3682.742273] Lustre: server umount lustre-MDT0000 complete [ 3682.784374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3682.786134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3682.786686] LustreError: Skipped 8 previous similar messages [ 3682.791278] Lustre: Skipped 47 previous similar messages [ 3697.242330] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3697.244287] LDISKFS-fs (dm-0): recovery complete [ 3697.247338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3697.285900] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3697.289116] LustreError: Skipped 10 previous similar messages [ 3698.938593] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3702.754677] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3702.757173] Lustre: Skipped 47 previous similar messages [ 3702.776984] Lustre: 6496:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3bf5936680 x1870757032235264/t270582939664(0) o36->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:680/0 lens 504/2880 e 0 to 0 dl 1784096635 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 [ 3702.781715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3203 to 0x280000401:3329) [ 3702.781889] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3297) [ 3707.132610] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 02:23:04 (1784096584) [ 3707.565834] Lustre: *** cfs_fail_loc=107, val=2147483648*** [ 3708.890928] Lustre: *** cfs_fail_loc=13b, val=315*** [ 3708.892563] LustreError: 6499:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3aca469880 x1870757032247552/t274877906960(0) o35->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:642/0 lens 392/456 e 0 to 0 dl 1784096597 ref 1 fl Interpret:/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3712.405397] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3713.046938] Lustre: Failing over lustre-MDT0000 [ 3713.136439] Lustre: server umount lustre-MDT0000 complete [ 3727.356111] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3727.357885] LDISKFS-fs (dm-0): recovery complete [ 3727.361219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.140623] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3728.143917] Lustre: Skipped 11 previous similar messages [ 3728.850834] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3732.969071] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 3732.972512] Lustre: Skipped 11 previous similar messages [ 3732.977661] Lustre: 6500:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3bf594ce00 x1870757032247552/t274877906960(0) o35->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:666/0 lens 392/456 e 0 to 0 dl 1784096621 ref 1 fl Interpret:/602/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 [ 3732.987719] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3329) [ 3732.987895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3331 to 0x280000401:3361) [ 3738.126886] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 02:23:35 (1784096615) [ 3738.506419] Lustre: *** cfs_fail_loc=12b, val=2147483991*** [ 3738.508579] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 3738.510652] Lustre: Skipped 1 previous similar message [ 3799.308543] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnecting [ 3799.310925] Lustre: Skipped 4 previous similar messages [ 3799.313582] Lustre: 9475:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3becc92d80 x1870757032258176/t279172874255(0) o101->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:21/0 lens 664/3488 e 0 to 0 dl 1784096731 ref 1 fl Interpret:/602/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 [ 3801.509607] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 02:24:38 (1784096678) [ 3804.121250] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3804.772134] Lustre: Failing over lustre-MDT0000 [ 3804.855174] Lustre: server umount lustre-MDT0000 complete [ 3819.107091] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 3819.108690] LDISKFS-fs (dm-0): recovery complete [ 3819.112423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.263222] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3819.266805] Lustre: Skipped 11 previous similar messages [ 3820.572852] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3824.640732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3363 to 0x280000401:3393) [ 3824.640926] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3361) [ 3826.586052] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3827.110760] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3841.155085] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 02:25:18 (1784096718) [ 3844.289717] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3844.936955] Lustre: Failing over lustre-MDT0000 [ 3845.075210] Lustre: server umount lustre-MDT0000 complete [ 3859.333266] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3859.334750] LDISKFS-fs (dm-0): recovery complete [ 3859.337767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3860.857530] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3864.577481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3395 to 0x280000401:3425) [ 3864.577824] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3267 to 0x2c0000401:3393) [ 3866.670565] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3867.367851] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3869.928636] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 3875.323912] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 02:25:52 (1784096752) [ 3887.497880] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3888.153353] Lustre: Failing over lustre-MDT0000 [ 3888.412128] Lustre: server umount lustre-MDT0000 complete [ 3902.837076] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3902.838516] LDISKFS-fs (dm-0): recovery complete [ 3902.841312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3904.273340] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3908.173830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4676 to 0x280000401:4705) [ 3908.173835] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4673) [ 3909.942978] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3910.475918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3924.881190] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 02:26:42 (1784096802) [ 3929.024350] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 3929.644851] Lustre: Failing over lustre-MDT0000 [ 3929.862473] Lustre: server umount lustre-MDT0000 complete [ 3931.406121] LustreError: 52308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.413314] LustreError: 52308:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 180 previous similar messages [ 3943.836867] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 3943.838270] LDISKFS-fs (dm-0): recovery complete [ 3943.840807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3943.904503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 3943.906244] Lustre: Skipped 3 previous similar messages [ 3945.333780] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 3949.052873] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4644 to 0x2c0000401:4705) [ 3949.052897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4707 to 0x280000401:4737) [ 3950.806962] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3951.367209] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3955.119748] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid 1475 0 [ 3955.626253] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 3957.697268] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 02:27:14 (1784096834) [ 3963.554077] Lustre: *** cfs_fail_loc=123, val=2147483648*** [ 4024.425110] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 4024.427534] LustreError: 11544:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bffa4ed80 x1870757035044864/t296352743435(0) o36->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:246/0 lens 66040/440 e 0 to 0 dl 1784096956 ref 1 fl Interpret:/600/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4024.436258] LustreError: 11544:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 4085.005876] Lustre: 11544:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3bc8eab050 x1870757035044864/t296352743435(0) o36->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:307/0 lens 66040/440 e 0 to 0 dl 1784097017 ref 1 fl Interpret:/602/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 projid:0 [ 4088.038642] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 4088.640012] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 02:29:25 (1784096965) [ 4092.691995] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4093.481876] Lustre: Failing over lustre-MDT0000 [ 4093.625126] Lustre: server umount lustre-MDT0000 complete [ 4107.644982] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4107.646589] LDISKFS-fs (dm-0): recovery complete [ 4107.649385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4107.749704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4107.752626] Lustre: Skipped 12 previous similar messages [ 4109.050853] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4113.025080] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4833) [ 4113.025080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4865) [ 4114.805871] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4115.329477] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4118.998509] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 02:29:56 (1784096996) [ 4124.430092] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 4127.318830] Lustre: Failing over lustre-OST0000 [ 4127.363719] Lustre: server umount lustre-OST0000 complete [ 4141.262092] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 4141.263189] LDISKFS-fs (dm-2): recovery complete [ 4141.265982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4143.172161] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4154.420868] Lustre: Failing over lustre-OST0000 [ 4154.457537] Lustre: server umount lustre-OST0000 complete [ 4167.284033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4169.337620] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4173.147630] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4173.680155] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4207.612294] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 02:31:24 (1784097084) [ 4208.516894] Lustre: Failing over lustre-MDT0000 [ 4208.609838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.795439] Lustre: server umount lustre-MDT0000 complete [ 4221.444297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4222.805802] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4227.061102] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4865) [ 4227.061104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4897) [ 4234.288191] Lustre: Failing over lustre-MDT0000 [ 4234.494289] Lustre: server umount lustre-MDT0000 complete [ 4247.354357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4248.859766] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4252.665552] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4806 to 0x2c0000401:4897) [ 4252.665600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4839 to 0x280000401:4929) [ 4254.376604] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4254.890681] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4258.263739] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 02:32:15 (1784097135) [ 4269.719960] Lustre: Failing over lustre-OST0000 [ 4269.778960] Lustre: server umount lustre-OST0000 complete [ 4282.663992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4284.632646] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4288.353467] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4288.924874] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4292.694702] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 02:32:49 (1784097169) [ 4293.335798] Lustre: Failing over lustre-MDT0000 [ 4293.601505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4293.602220] Lustre: server umount lustre-MDT0000 complete [ 4293.604525] LustreError: Skipped 9 previous similar messages [ 4293.608547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4293.613696] Lustre: Skipped 40 previous similar messages [ 4296.898793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4296.933452] Lustre: *** cfs_fail_loc=605, val=0*** [ 4296.934611] LustreError: 164405:0:(llog_obd.c:192:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffc092e100 failed: rc = -95 [ 4296.937511] LustreError: 164405:0:(obd_config.c:845:class_setup()) setup MGS failed (-95) [ 4296.939560] LustreError: 164405:0:(obd_mount.c:259:lustre_start_simple()) MGS setup error -95 [ 4296.942706] LustreError: 164405:0:(tgt_mount.c:116:server_deregister_mount()) MGS not registered [ 4296.944560] LustreError: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 4296.946196] LustreError: 164405:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 4296.950790] Lustre: server umount lustre-MDT0000 complete [ 4296.952226] LustreError: 164405:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 4299.674958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.714349] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4299.719082] LustreError: Skipped 8 previous similar messages [ 4301.229899] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4303.900284] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 02:33:01 (1784097181) [ 4304.867770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4304.869711] Lustre: Skipped 41 previous similar messages [ 4304.886938] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4899 to 0x2c0000401:4929) [ 4304.886956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4931 to 0x280000401:4961) [ 4307.218137] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4308.181948] Lustre: Failing over lustre-MDT0000 [ 4308.358146] Lustre: server umount lustre-MDT0000 complete [ 4322.540151] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4322.542881] LDISKFS-fs (dm-0): recovery complete [ 4322.546581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4324.111062] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4325.651480] Lustre: *** cfs_fail_loc=707, val=0*** [ 4386.066792] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 4386.080226] Lustre: 166572:0:(ldlm_lib.c:2084:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 180, extend: 1 [ 4386.083238] Lustre: 166572:0:(ldlm_lib.c:2084:extend_recovery_timer()) Skipped 1 previous similar message [ 4386.151154] Lustre: lustre-MDT0000: Recovery over after 1:01, of 2 clients 2 recovered and 0 were evicted. [ 4386.154105] Lustre: Skipped 11 previous similar messages [ 4386.167232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4974 to 0x280000401:4993) [ 4386.167285] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:4943 to 0x2c0000401:4961) [ 4387.992959] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4388.483297] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4392.209235] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 02:34:29 (1784097269) [ 4415.619111] LustreError: 9475:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9b3bc6c34a80 x1870757035923456/t0(0) o101->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:594/0 lens 664/0 e 0 to 0 dl 1784097304 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4415.626482] LustreError: 9475:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 11000ms [ 4426.656134] LustreError: 9475:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4426.662401] LustreError: 6500:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9b3bfcf65f80 x1870757035924736/t0(0) o35->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:605/0 lens 392/0 e 0 to 0 dl 1784097315 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4431.841796] LustreError: 43777:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9b3ac803e300 x1870757042918656/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:610/0 lens 544/0 e 0 to 0 dl 1784097320 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 projid:4294967295 [ 4431.848837] LustreError: 43777:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 48 previous similar messages [ 4438.326790] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 02:35:15 (1784097315) [ 4461.493685] LustreError: 42616:0:(tgt_handler.c:2843:tgt_brw_write()) cfs_fail_timeout id 224 sleeping for 11000ms [ 4472.520082] LustreError: 42616:0:(tgt_handler.c:2843:tgt_brw_write()) cfs_fail_timeout id 224 awake [ 4475.565670] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 02:35:52 (1784097352) [ 4498.494734] LustreError: 11544:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9b3bf58be300 x1870757035947392/t0(0) o101->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:676/0 lens 576/0 e 0 to 0 dl 1784097386 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4498.502359] LustreError: 11544:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 5000ms [ 4503.600106] LustreError: 11544:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4503.605227] LustreError: 6497:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9b3acb115c00 x1870757035948160/t0(0) o101->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:683/0 lens 664/0 e 0 to 0 dl 1784097393 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4514.273918] LustreError: 43774:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9b3bf8261f80 x1870757042960000/t0(0) o6->lustre-MDT0000-mdtlov_UUID@0@lo:692/0 lens 544/0 e 0 to 0 dl 1784097402 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'osp-syn-1-0.0' uid:0 gid:0 projid:4294967295 [ 4514.284148] LustreError: 43774:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 118 previous similar messages [ 4514.408114] LustreError: 52308:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4526.237395] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 02:36:43 (1784097403) [ 4611.037597] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 02:38:08 (1784097488) [ 4633.874914] LustreError: 6497:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9b3becc5a300 x1870757036023552/t0(0) o101->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:57/0 lens 576/0 e 0 to 0 dl 1784097522 ref 1 fl Interpret:/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 [ 4633.883650] LustreError: 6497:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 99 previous similar messages [ 4633.886505] LustreError: 6497:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 4633.890112] LustreError: 6497:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 1 previous similar message [ 4634.312107] LustreError: 6497:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4650.130025] LustreError: 52308:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a sleeping for 400ms [ 4650.133413] LustreError: 52308:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 4650.552138] LustreError: 52308:0:(service.c:2559:ptlrpc_server_handle_request()) cfs_fail_timeout id 50a awake [ 4650.554706] LustreError: 52308:0:(service.c:2559:ptlrpc_server_handle_request()) Skipped 38 previous similar messages [ 4665.970064] LustreError: 6496:0:(service.c:2558:ptlrpc_server_handle_request()) @@@ HIT req@ffff9b3acb115880 x1870757036042240/t0(0) o36->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:89/0 lens 504/0 e 0 to 0 dl 1784097554 ref 1 fl Interpret:/200/ffffffff rc 0/-1 job:'unlinkmany.0' uid:0 gid:0 projid:4294967295 [ 4665.977402] LustreError: 6496:0:(service.c:2558:ptlrpc_server_handle_request()) Skipped 80 previous similar messages [ 4677.940424] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 02:39:15 (1784097555) [ 4702.457304] Lustre: DEBUG MARKER: phase 2 [ 4705.475687] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 02:39:42 (1784097582) [ 4775.440992] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 02:40:52 (1784097652) [ 4775.988707] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1 [ 4776.591680] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 02:40:53 (1784097653) [ 4778.465514] Lustre: DEBUG MARKER: Started rundbench load pid=135015 ... [ 4781.964756] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4783.473036] Lustre: DEBUG MARKER: test_70b fail mds1 1 times [ 4784.094751] Lustre: Failing over lustre-MDT0000 [ 4784.347823] Lustre: server umount lustre-MDT0000 complete [ 4785.632716] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4785.639088] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 88 previous similar messages [ 4798.121585] LDISKFS-fs (dm-0): 4 truncates cleaned up [ 4798.122615] LDISKFS-fs (dm-0): recovery complete [ 4798.125364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4798.229239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4798.231505] Lustre: Skipped 7 previous similar messages [ 4798.241306] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4798.243146] Lustre: Skipped 11 previous similar messages [ 4799.424484] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4801.804627] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4801.808156] Lustre: Skipped 12 previous similar messages [ 4803.878359] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5088 to 0x2c0000401:5121) [ 4803.878406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5125 to 0x280000401:5153) [ 4805.812307] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4806.396177] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4811.532054] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4813.058124] Lustre: DEBUG MARKER: test_70b fail mds2 2 times [ 4813.741302] Lustre: Failing over lustre-MDT0001 [ 4813.792895] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4813.795038] Lustre: Skipped 5 previous similar messages [ 4813.899264] Lustre: server umount lustre-MDT0001 complete [ 4828.166246] LDISKFS-fs (dm-1): 8 truncates cleaned up [ 4828.168146] LDISKFS-fs (dm-1): recovery complete [ 4828.177700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.662842] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4833.781874] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:449) [ 4833.781912] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:449) [ 4835.571528] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4836.116479] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4840.954461] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4842.483277] Lustre: DEBUG MARKER: test_70b fail mds1 3 times [ 4843.182087] Lustre: Failing over lustre-MDT0000 [ 4843.197702] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 4843.353667] Lustre: server umount lustre-MDT0000 complete [ 4857.246107] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4857.247935] LDISKFS-fs (dm-0): recovery complete [ 4857.251794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.624253] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4862.770178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5538 to 0x280000401:5569) [ 4862.770186] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5505 to 0x2c0000401:5537) [ 4864.559203] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4865.084303] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4869.980932] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 4871.491586] Lustre: DEBUG MARKER: test_70b fail mds2 4 times [ 4872.141715] Lustre: Failing over lustre-MDT0001 [ 4872.278622] Lustre: server umount lustre-MDT0001 complete [ 4886.296714] LDISKFS-fs (dm-1): 5 truncates cleaned up [ 4886.298772] LDISKFS-fs (dm-1): recovery complete [ 4886.303280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4887.759253] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4891.639503] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:421 to 0x280000400:481) [ 4891.639551] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:403 to 0x2c0000400:481) [ 4893.443351] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4893.951385] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4898.801600] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 4900.326030] Lustre: DEBUG MARKER: test_70b fail mds1 5 times [ 4900.928648] Lustre: Failing over lustre-MDT0000 [ 4900.938693] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 4901.186146] Lustre: server umount lustre-MDT0000 complete [ 4901.856484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4901.856518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.858473] LustreError: Skipped 5 previous similar messages [ 4901.863176] Lustre: Skipped 22 previous similar messages [ 4915.335478] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 4915.336781] LDISKFS-fs (dm-0): recovery complete [ 4915.339394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4915.375064] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4915.378415] LustreError: Skipped 3 previous similar messages [ 4916.811937] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 4920.802286] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 4920.805613] Lustre: Skipped 21 previous similar messages [ 4921.085301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5988 to 0x280000401:6017) [ 4921.085351] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5956 to 0x2c0000401:5985) [ 4923.002920] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4923.575489] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4926.968570] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 02:43:24 (1784097804) [ 5049.836638] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5060.531465] Lustre: DEBUG MARKER: test_70c fail mds2 1 times [ 5061.131092] Lustre: Failing over lustre-MDT0001 [ 5061.902503] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 5066.833086] Lustre: server umount lustre-MDT0001 complete [ 5081.195920] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5081.197231] LDISKFS-fs (dm-1): recovery complete [ 5081.202420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5082.759791] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5089.014150] Lustre: lustre-MDT0001: Recovery over after 0:07, of 2 clients 2 recovered and 0 were evicted. [ 5089.018327] Lustre: Skipped 5 previous similar messages [ 5089.046020] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:1795 to 0x280000400:1825) [ 5089.050427] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:1794 to 0x2c0000400:1825) [ 5091.081214] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5091.592397] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5215.561154] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5226.433725] Lustre: DEBUG MARKER: test_70c fail mds2 2 times [ 5227.126556] Lustre: Failing over lustre-MDT0001 [ 5227.142765] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5227.144607] Lustre: Skipped 3 previous similar messages [ 5227.394707] Lustre: server umount lustre-MDT0001 complete [ 5241.555987] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5241.557536] LDISKFS-fs (dm-1): recovery complete [ 5241.562233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5243.071439] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5248.951315] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:3220 to 0x2c0000400:3265) [ 5248.951578] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3220 to 0x280000400:3265) [ 5250.976863] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5251.520599] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5272.465286] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 02:49:09 (1784098149) [ 5395.374134] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5406.063743] Lustre: DEBUG MARKER: test_70d fail mds1 1 times [ 5406.684714] Lustre: Failing over lustre-MDT0000 [ 5406.700803] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 5406.942345] Lustre: server umount lustre-MDT0000 complete [ 5408.744280] LustreError: 7897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784098286 with bad export cookie 9498434355224486970 [ 5408.749529] LustreError: 6502:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.754633] LustreError: 6502:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 86 previous similar messages [ 5420.862346] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5420.863706] LDISKFS-fs (dm-0): recovery complete [ 5420.866590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5420.960433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5420.962389] Lustre: Skipped 6 previous similar messages [ 5420.971287] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5420.974904] Lustre: Skipped 6 previous similar messages [ 5422.254436] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5422.348439] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5422.351436] Lustre: Skipped 6 previous similar messages [ 5427.085422] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9406 to 0x2c0000401:9441) [ 5427.085567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9438 to 0x280000401:9473) [ 5428.914700] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5429.445197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5553.363399] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5564.081382] Lustre: DEBUG MARKER: test_70d fail mds1 2 times [ 5564.694433] Lustre: Failing over lustre-MDT0000 [ 5564.695056] LustreError: lustre-MDT0000-osp-MDT0001: operation ldlm_cancel to node 0@lo failed: rc = -19 [ 5564.698541] LustreError: Skipped 2 previous similar messages [ 5564.700102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5564.704238] Lustre: Skipped 12 previous similar messages [ 5564.705854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5564.979641] Lustre: server umount lustre-MDT0000 complete [ 5579.100081] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5579.102042] LDISKFS-fs (dm-0): recovery complete [ 5579.105917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5579.150958] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5579.154951] LustreError: Skipped 1 previous similar message [ 5580.624171] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5584.354852] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5584.357939] Lustre: Skipped 13 previous similar messages [ 5585.372914] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:9870 to 0x2c0000401:9889) [ 5585.372920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:9902 to 0x280000401:9921) [ 5587.439385] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5588.050458] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5591.637255] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 02:54:28 (1784098468) [ 5714.935708] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 5725.667717] Lustre: DEBUG MARKER: test_70e fail mds2 1 times [ 5726.248178] Lustre: Failing over lustre-MDT0001 [ 5726.255740] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5726.257940] Lustre: Skipped 1 previous similar message [ 5726.512133] Lustre: server umount lustre-MDT0001 complete [ 5740.465512] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 5740.466993] LDISKFS-fs (dm-1): recovery complete [ 5740.470666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5741.830339] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5745.640088] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5745.644496] Lustre: Skipped 3 previous similar messages [ 5745.662309] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4267 to 0x280000400:4289) [ 5745.662372] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4267 to 0x2c0000400:4289) [ 5747.673805] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5748.322670] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5872.661183] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 5883.416634] Lustre: DEBUG MARKER: test_70e fail mds1 2 times [ 5884.048730] Lustre: Failing over lustre-MDT0000 [ 5884.191766] Lustre: server umount lustre-MDT0000 complete [ 5898.108439] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 5898.109619] LDISKFS-fs (dm-0): recovery complete [ 5898.112196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5899.507353] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5906.271540] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13857 to 0x2c0000401:13889) [ 5906.271835] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13888 to 0x280000401:13921) [ 5908.202234] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5908.842703] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5912.423339] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 02:59:49 (1784098789) [ 5918.488459] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5919.973032] Lustre: DEBUG MARKER: test_70f failing OST 1 times [ 5920.576349] Lustre: Failing over lustre-OST0000 [ 5920.628035] Lustre: server umount lustre-OST0000 complete [ 5934.908721] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5934.910913] LDISKFS-fs (dm-2): recovery complete [ 5934.914882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5936.912926] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5940.995872] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5941.678249] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5950.595589] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 5952.092495] Lustre: DEBUG MARKER: test_70f failing OST 2 times [ 5952.760547] Lustre: Failing over lustre-OST0000 [ 5952.804590] Lustre: server umount lustre-OST0000 complete [ 5966.915531] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 5966.917508] LDISKFS-fs (dm-2): recovery complete [ 5966.921861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.862939] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 5972.992335] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5973.649409] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5979.259847] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 03:00:56 (1784098856) [ 6102.351218] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6105.072383] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6115.874599] Lustre: DEBUG MARKER: fail mds1 mds2 1 times [ 6116.541383] Lustre: Failing over lustre-MDT0000 [ 6116.602170] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 6116.605962] Lustre: Skipped 1 previous similar message [ 6122.884219] Lustre: server umount lustre-MDT0000 complete [ 6124.116659] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784099001 with bad export cookie 9498434355228579359 [ 6124.117766] Lustre: Failing over lustre-MDT0001 [ 6124.120449] LustreError: 6483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6124.363267] Lustre: server umount lustre-MDT0001 complete [ 6139.326519] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6139.329797] LDISKFS-fs (dm-1): recovery complete [ 6139.336567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.389064] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6139.390347] LDISKFS-fs (dm-0): recovery complete [ 6139.393294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.424747] LustreError: 200019:0:(llog.c:1655:llog_backup()) MGC192.168.201.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6139.429559] Lustre: 200019:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6141.197979] LustreError: 200026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.203165] LustreError: 200026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 62 previous similar messages [ 6142.432126] Lustre: 3630:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784099003/real 1784099003] req@ffff9b3ad16eb800 x1870757086693888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784099019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6142.441804] Lustre: 3630:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6148.576418] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3bfdb74380 x1870757086696832/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6148.656820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6148.659239] Lustre: Skipped 5 previous similar messages [ 6148.667486] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6148.671727] Lustre: Skipped 5 previous similar messages [ 6149.775074] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6149.778028] Lustre: Skipped 6 previous similar messages [ 6150.160746] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6150.294243] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6154.805109] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4291 to 0x280000400:4321) [ 6154.805234] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4291 to 0x2c0000400:4321) [ 6159.645592] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13963 to 0x2c0000401:13985) [ 6159.645820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:13994 to 0x280000401:14017) [ 6161.580664] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6162.228299] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6162.855911] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6166.460702] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 03:04:03 (1784099043) [ 6169.032839] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6169.943336] Lustre: Failing over lustre-MDT0000 [ 6170.080569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6170.084060] LustreError: Skipped 4 previous similar messages [ 6170.086065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6170.091502] Lustre: Skipped 20 previous similar messages [ 6170.167681] Lustre: server umount lustre-MDT0000 complete [ 6184.215658] LDISKFS-fs (dm-0): 3 truncates cleaned up [ 6184.217541] LDISKFS-fs (dm-0): recovery complete [ 6184.220464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.253847] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6184.257021] LustreError: Skipped 2 previous similar messages [ 6185.230109] Lustre: *** cfs_fail_loc=302, val=2147483648*** [ 6185.634979] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6189.538044] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6189.539978] Lustre: Skipped 20 previous similar messages [ 6200.596771] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 6200.629270] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13963 to 0x2c0000401:14017) [ 6200.629275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14019 to 0x280000401:14049) [ 6202.569036] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6203.095589] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6206.574984] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 03:04:43 (1784099083) [ 6209.067114] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6210.030157] Lustre: Failing over lustre-MDT0000 [ 6210.218873] Lustre: server umount lustre-MDT0000 complete [ 6224.304441] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6224.305724] LDISKFS-fs (dm-0): recovery complete [ 6224.308029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6225.706554] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6226.198343] Lustre: *** cfs_fail_loc=157, val=2147483648*** [ 6226.199793] LustreError: 200024:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3ad02a9180 x1870757127723008/t343597384421(343597384421) o101->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:139/0 lens 520/664 e 0 to 0 dl 1784099114 ref 1 fl Interpret:/604/0 rc 301/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6241.556434] Lustre: lustre-MDT0000: Client 9d96a3b9-8c31-46b4-b224-2e30ff5b404a (at 192.168.201.12@tcp) reconnected, waiting for 2 clients in recovery for 0:54 [ 6241.561044] Lustre: 200051:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3ad14cce00 x1870757127723008/t343597384421(343597384421) o101->9d96a3b9-8c31-46b4-b224-2e30ff5b404a@192.168.201.12@tcp:155/0 lens 520/3488 e 0 to 0 dl 1784099130 ref 1 fl Interpret:/606/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 [ 6241.591859] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13963 to 0x2c0000401:14049) [ 6241.591899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14081) [ 6243.580808] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6244.150543] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6247.544201] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 03:05:24 (1784099124) [ 6249.135360] Lustre: Failing over lustre-OST0000 [ 6249.179234] Lustre: server umount lustre-OST0000 complete [ 6250.431222] Lustre: Failing over lustre-MDT0000 [ 6250.516487] Lustre: server umount lustre-MDT0000 complete [ 6263.296320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6264.696703] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6265.430114] Lustre: lustre-MDT0000: Denying connection for new client fc790c88-3fb0-4a73-ad90-cb2e5c02ba96 (at 192.168.201.12@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 6268.914787] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:13963 to 0x2c0000401:14081) [ 6272.611396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6274.595575] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6278.118895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14113) [ 6278.681391] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 03:05:55 (1784099155) [ 6279.024149] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6279.025529] LustreError: 8421:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bffb40000 x1870757088174336/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:192/0 lens 1312/4320 e 0 to 0 dl 1784099167 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6281.384206] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6282.012465] Lustre: Failing over lustre-MDT0000 [ 6282.350879] Lustre: server umount lustre-MDT0000 complete [ 6295.520118] Lustre: 200056:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784099156/real 1784099156] req@ffff9b3bffb42a00 x1870757088174336/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 1312/4320 e 0 to 1 dl 1784099172 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6295.528435] Lustre: 200056:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [ 6296.280909] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6296.282078] LDISKFS-fs (dm-0): recovery complete [ 6296.284904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6297.690475] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6301.687410] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14113) [ 6301.687426] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14145) [ 6303.474271] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6303.976400] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6307.450523] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 03:06:24 (1784099184) [ 6307.800809] LustreError: 8421:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bed269f80 x1870757088204928/t365072220172(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:221/0 lens 2520/4320 e 0 to 0 dl 1784099196 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6310.274854] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6312.763635] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6313.414325] Lustre: Failing over lustre-MDT0001 [ 6319.154490] Lustre: server umount lustre-MDT0001 complete [ 6333.262732] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6333.264122] LDISKFS-fs (dm-1): recovery complete [ 6333.268669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6334.801193] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6338.545175] Lustre: 200047:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3bffb39c00 x1870757134752768/t30064773421(0) o36->fc790c88-3fb0-4a73-ad90-cb2e5c02ba96@192.168.201.12@tcp:274/0 lens 560/2880 e 0 to 0 dl 1784099249 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6338.552326] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4332 to 0x2c0000400:4353) [ 6338.552338] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4332 to 0x280000400:4353) [ 6340.347668] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6340.832076] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6344.364619] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 03:07:01 (1784099221) [ 6344.715747] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6344.717208] Lustre: Skipped 1 previous similar message [ 6344.718278] LustreError: 182159:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3aca208a80 x1870757088238848/t365072220193(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:258/0 lens 2520/4320 e 0 to 0 dl 1784099233 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295 [ 6347.280717] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6349.809182] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6350.445777] Lustre: Failing over lustre-MDT0000 [ 6350.705930] Lustre: server umount lustre-MDT0000 complete [ 6364.876880] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6364.878972] LDISKFS-fs (dm-0): recovery complete [ 6364.882707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.579405] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6370.319969] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6370.322277] Lustre: Skipped 11 previous similar messages [ 6370.337438] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14145) [ 6370.337438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14177) [ 6372.166745] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6372.723496] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6374.675324] Lustre: Failing over lustre-MDT0001 [ 6374.777561] Lustre: server umount lustre-MDT0001 complete [ 6389.120328] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6389.121611] LDISKFS-fs (dm-1): recovery complete [ 6389.125364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.594995] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6394.357612] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4364 to 0x2c0000400:4385) [ 6394.357641] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4364 to 0x280000400:4385) [ 6396.069443] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6396.552108] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6400.071205] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 03:07:57 (1784099277) [ 6405.890396] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6408.414090] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6409.062305] Lustre: Failing over lustre-MDT0000 [ 6409.362303] Lustre: server umount lustre-MDT0000 complete [ 6410.645048] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784099288 with bad export cookie 9498434355229609647 [ 6410.645970] Lustre: Failing over lustre-MDT0001 [ 6410.648633] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6415.841096] LustreError: 216528:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9b3bf762ad80 x1870757088302720/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6415.846866] LustreError: 216528:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6415.968531] Lustre: server umount lustre-MDT0001 complete [ 6431.162802] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6431.166633] LDISKFS-fs (dm-1): recovery complete [ 6431.171799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.176745] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6431.178286] LDISKFS-fs (dm-0): recovery complete [ 6431.182452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.277264] LustreError: 217623:0:(llog.c:1655:llog_backup()) MGC192.168.201.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6431.279690] Lustre: 217623:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6436.320703] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3bffb39f80 x1870757088303872/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6437.941554] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6438.115258] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6450.976622] Lustre: 217637:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3bfcfb8a80 x1870757134820736/t38654705732(0) o36->fc790c88-3fb0-4a73-ad90-cb2e5c02ba96@192.168.201.12@tcp:390/0 lens 560/2880 e 0 to 0 dl 1784099365 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6451.033449] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4396 to 0x2c0000400:4417) [ 6451.033594] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4396 to 0x280000400:4417) [ 6451.043061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14209) [ 6451.043290] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14177) [ 6453.025716] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6453.546365] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6454.083315] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6457.538386] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 03:08:54 (1784099334) [ 6457.870548] LustreError: 217638:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bfd947100 x1870757134852992/t42949673028(0) o36->fc790c88-3fb0-4a73-ad90-cb2e5c02ba96@192.168.201.12@tcp:410/0 lens 560/448 e 0 to 0 dl 1784099385 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6457.878042] LustreError: 217638:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 1 previous similar message [ 6463.425866] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6464.063133] Lustre: Failing over lustre-MDT0000 [ 6464.344855] Lustre: server umount lustre-MDT0000 complete [ 6478.351550] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6478.353013] LDISKFS-fs (dm-0): recovery complete [ 6478.356264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6479.792029] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6483.963590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14241) [ 6483.963642] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14209) [ 6485.715633] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6486.190723] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6512.908415] Lustre: lustre-MDT0001: Client fc790c88-3fb0-4a73-ad90-cb2e5c02ba96 (at 192.168.201.12@tcp) reconnecting [ 6512.912262] Lustre: Skipped 2 previous similar messages [ 6512.914795] Lustre: 217636:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3bf701ad80 x1870757134852992/t42949673028(0) o36->fc790c88-3fb0-4a73-ad90-cb2e5c02ba96@192.168.201.12@tcp:465/0 lens 560/2880 e 0 to 0 dl 1784099440 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6515.343454] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 03:09:52 (1784099392) [ 6515.685963] Lustre: *** cfs_fail_loc=119, val=2147483648*** [ 6515.688057] Lustre: Skipped 2 previous similar messages [ 6518.210300] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6518.864398] Lustre: Failing over lustre-MDT0001 [ 6518.966777] Lustre: server umount lustre-MDT0001 complete [ 6533.257209] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6533.258907] LDISKFS-fs (dm-1): recovery complete [ 6533.264852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.751985] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6538.754656] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4438 to 0x280000400:4481) [ 6538.754714] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4438 to 0x2c0000400:4481) [ 6540.454687] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6540.945222] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6544.468924] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 03:10:21 (1784099421) [ 6550.340896] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6552.769774] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6553.404345] Lustre: Failing over lustre-MDT0000 [ 6553.664476] Lustre: server umount lustre-MDT0000 complete [ 6561.039137] Lustre: 217676:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3bfce1ce00 x1870757134915456/t47244640324(0) o36->fc790c88-3fb0-4a73-ad90-cb2e5c02ba96@192.168.201.12@tcp:474/0 lens 560/2880 e 0 to 0 dl 1784099449 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6561.047159] Lustre: 217676:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6567.795902] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6567.797104] LDISKFS-fs (dm-0): recovery complete [ 6567.799451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6569.232116] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6573.071957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14273) [ 6573.072055] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14241) [ 6574.790919] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6575.272752] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6577.427156] Lustre: Failing over lustre-MDT0001 [ 6577.528464] Lustre: server umount lustre-MDT0001 complete [ 6591.682361] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6591.684210] LDISKFS-fs (dm-1): recovery complete [ 6591.688712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.154683] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6597.107880] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4492 to 0x280000400:4513) [ 6597.107880] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4492 to 0x2c0000400:4513) [ 6598.836105] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6599.336111] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6602.809858] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 03:11:19 (1784099479) [ 6603.164425] LustreError: 218420:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bf701ad80 x1870757134952832/t51539607620(0) o36->fc790c88-3fb0-4a73-ad90-cb2e5c02ba96@192.168.201.12@tcp:516/0 lens 560/448 e 0 to 0 dl 1784099491 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6603.170414] LustreError: 218420:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 2 previous similar messages [ 6608.602217] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6611.109562] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6611.983426] Lustre: Failing over lustre-MDT0000 [ 6612.232942] Lustre: server umount lustre-MDT0000 complete [ 6613.416750] LustreError: 9499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784099490 with bad export cookie 9498434355229623381 [ 6613.417559] Lustre: Failing over lustre-MDT0001 [ 6613.420261] LustreError: 9499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6613.522128] Lustre: server umount lustre-MDT0001 complete [ 6628.394053] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6628.395673] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6628.396433] LDISKFS-fs (dm-1): recovery complete [ 6628.397836] LDISKFS-fs (dm-0): recovery complete [ 6628.399547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.402816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.495796] LustreError: 228821:0:(llog.c:1655:llog_backup()) MGC192.168.201.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6628.498161] Lustre: 228821:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6633.952274] Lustre: 3628:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784099495/real 1784099495] req@ffff9b3aca209500 x1870757088475776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784099511 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6633.959407] Lustre: 3628:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6638.048781] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3aca20a300 x1870757088478464/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6639.703454] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6639.840360] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6644.273526] Lustre: 228837:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3bf701aa00 x1870757134952832/t51539607620(0) o36->fc790c88-3fb0-4a73-ad90-cb2e5c02ba96@192.168.201.12@tcp:557/0 lens 560/2880 e 0 to 0 dl 1784099532 ref 1 fl Interpret:/202/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 [ 6644.279380] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4524 to 0x2c0000400:4545) [ 6644.279688] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4524 to 0x280000400:4545) [ 6644.319908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14305) [ 6644.319939] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14273) [ 6646.126763] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6646.626022] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6647.134833] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6650.785242] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 03:12:07 (1784099527) [ 6653.746900] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6654.427145] Lustre: Failing over lustre-MDT0001 [ 6654.433675] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6654.435688] Lustre: Skipped 19 previous similar messages [ 6660.148421] Lustre: server umount lustre-MDT0001 complete [ 6674.271457] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6674.273135] LDISKFS-fs (dm-1): recovery complete [ 6674.277339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6675.762445] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6679.543383] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4556 to 0x2c0000400:4577) [ 6679.543408] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4556 to 0x280000400:4577) [ 6681.345500] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6681.893583] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6685.408857] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 03:12:42 (1784099562) [ 6688.828471] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6689.454218] Lustre: Failing over lustre-MDT0000 [ 6689.829102] Lustre: server umount lustre-MDT0000 complete [ 6703.982707] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6703.984220] LDISKFS-fs (dm-0): recovery complete [ 6703.987940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.567184] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6709.249077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14305) [ 6709.249113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14337) [ 6711.054412] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6711.608451] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6714.922321] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 03:13:12 (1784099592) [ 6717.880740] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6720.441928] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6721.078897] Lustre: Failing over lustre-MDT0000 [ 6721.417717] Lustre: server umount lustre-MDT0000 complete [ 6735.487728] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6735.489700] LDISKFS-fs (dm-0): recovery complete [ 6735.492628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.991023] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6740.993108] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14337) [ 6740.993109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14369) [ 6742.933422] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6743.461321] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6745.494677] Lustre: Failing over lustre-MDT0001 [ 6745.611382] Lustre: server umount lustre-MDT0001 complete [ 6746.081802] LustreError: 182160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6746.085785] LustreError: 182160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 227 previous similar messages [ 6759.585556] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6759.586902] LDISKFS-fs (dm-1): recovery complete [ 6759.590538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.687974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6759.691574] Lustre: Skipped 20 previous similar messages [ 6759.700542] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6759.702543] Lustre: Skipped 20 previous similar messages [ 6760.911253] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6760.914145] Lustre: Skipped 19 previous similar messages [ 6760.988217] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6765.042988] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4556 to 0x280000400:4609) [ 6765.043022] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4556 to 0x2c0000400:4609) [ 6766.978450] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6767.605365] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6771.281717] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 03:14:08 (1784099648) [ 6771.766224] Lustre: *** cfs_fail_loc=1701, val=2147483648*** [ 6771.768425] Lustre: Skipped 5 previous similar messages [ 6774.634562] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6778.318870] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6779.158733] Lustre: Failing over lustre-MDT0000 [ 6779.425407] Lustre: server umount lustre-MDT0000 complete [ 6780.385358] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.386217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6780.392524] Lustre: Skipped 68 previous similar messages [ 6780.396484] LustreError: Skipped 18 previous similar messages [ 6781.084141] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784099658 with bad export cookie 9498434355229633790 [ 6781.085927] Lustre: Failing over lustre-MDT0001 [ 6781.091933] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6787.041475] LustreError: 238874:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9b3bfd85e680 x1870757088619136/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 projid:4294967295 [ 6787.049579] LustreError: 238874:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000401:0x1:0x0]: rc = -5 [ 6787.166670] Lustre: server umount lustre-MDT0001 complete [ 6802.969839] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6802.973043] LDISKFS-fs (dm-0): recovery complete [ 6802.974052] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6802.977879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6802.979099] LDISKFS-fs (dm-1): recovery complete [ 6802.990569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6805.984556] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3bfe095500 x1870757088619904/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6807.808479] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6807.878488] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6811.427982] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 6811.432241] Lustre: Skipped 66 previous similar messages [ 6812.399098] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14369) [ 6812.400337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14401) [ 6812.421664] Lustre: 240759:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b3ad06eb100 x1870757135029760/t64424509442(0) o36->fc790c88-3fb0-4a73-ad90-cb2e5c02ba96@192.168.201.12@tcp:750/0 lens 496/2888 e 0 to 0 dl 1784099725 ref 1 fl Interpret:/202/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6812.425708] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4556 to 0x280000400:4641) [ 6812.428131] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4556 to 0x2c0000400:4641) [ 6812.433518] Lustre: 240759:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message [ 6814.629878] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6815.345061] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6815.841231] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6819.340498] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 03:14:56 (1784099696) [ 6823.085407] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6823.693406] Lustre: Failing over lustre-MDT0000 [ 6823.840369] Lustre: server umount lustre-MDT0000 complete [ 6835.980827] Lustre: lustre-MDT0001: Client fc790c88-3fb0-4a73-ad90-cb2e5c02ba96 (at 192.168.201.12@tcp) reconnecting [ 6835.984416] Lustre: Skipped 1 previous similar message [ 6838.360198] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6838.362359] LDISKFS-fs (dm-0): recovery complete [ 6838.366346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6838.409927] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6838.413419] LustreError: Skipped 11 previous similar messages [ 6840.287710] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6843.909263] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14401) [ 6843.909416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14433) [ 6845.721767] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6846.274692] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6849.770099] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 03:15:26 (1784099726) [ 6852.995521] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6853.649160] Lustre: Failing over lustre-MDT0001 [ 6853.756888] Lustre: server umount lustre-MDT0001 complete [ 6867.819200] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6867.820453] LDISKFS-fs (dm-1): recovery complete [ 6867.824535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6869.389745] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6873.080940] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4556 to 0x280000400:4673) [ 6873.080946] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4556 to 0x2c0000400:4673) [ 6874.866493] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6875.411225] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6878.704605] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 03:15:55 (1784099755) [ 6879.037756] LustreError: 240759:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ffff9b3bfdcf6d80 x1870757135066880/t73014444034(0) o36->fc790c88-3fb0-4a73-ad90-cb2e5c02ba96@192.168.201.12@tcp:37/0 lens 496/456 e 0 to 0 dl 1784099767 ref 1 fl Interpret:/200/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:4294967295 [ 6879.044751] LustreError: 240759:0:(ldlm_lib.c:3345:target_send_reply_msg()) Skipped 6 previous similar messages [ 6881.667980] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6884.442889] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6885.028194] Lustre: Failing over lustre-MDT0000 [ 6885.185892] Lustre: server umount lustre-MDT0000 complete [ 6899.153270] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6899.155236] LDISKFS-fs (dm-0): recovery complete [ 6899.159216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.598988] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6904.316534] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14433) [ 6904.316537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14465) [ 6906.633448] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6907.302810] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6909.657304] Lustre: Failing over lustre-MDT0001 [ 6909.780824] Lustre: server umount lustre-MDT0001 complete [ 6924.746356] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6924.747536] LDISKFS-fs (dm-1): recovery complete [ 6924.751078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6926.213963] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6929.914467] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4556 to 0x280000400:4705) [ 6929.914494] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4556 to 0x2c0000400:4705) [ 6932.034828] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6932.599248] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6936.238228] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 03:16:53 (1784099813) [ 6940.105170] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 6943.043191] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 [ 6943.749270] Lustre: Failing over lustre-MDT0000 [ 6943.959846] Lustre: server umount lustre-MDT0000 complete [ 6945.350925] LustreError: 151745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784099822 with bad export cookie 9498434355229641350 [ 6945.352369] Lustre: Failing over lustre-MDT0001 [ 6945.355404] LustreError: 151745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6945.461357] Lustre: server umount lustre-MDT0001 complete [ 6960.382312] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 6960.384477] LDISKFS-fs (dm-0): recovery complete [ 6960.385421] LDISKFS-fs (dm-1): 6 truncates cleaned up [ 6960.387911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.388497] LDISKFS-fs (dm-1): recovery complete [ 6960.396262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.486958] LustreError: 251166:0:(llog.c:1655:llog_backup()) MGC192.168.201.112@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6960.489714] Lustre: 251166:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.112@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6965.728209] Lustre: 3630:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784099827/real 1784099827] req@ffff9b3bf7019180 x1870757088758528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1784099843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6965.735584] Lustre: 3630:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [ 6969.825216] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3ad12d6680 x1870757088760960/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 6971.633362] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6971.673071] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 6975.487205] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6975.491969] Lustre: Skipped 19 previous similar messages [ 6975.543282] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14083 to 0x2c0000401:14465) [ 6975.543376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14051 to 0x280000401:14497) [ 6975.552282] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:4556 to 0x280000400:4737) [ 6975.552408] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4556 to 0x2c0000400:4737) [ 6978.085264] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6978.720234] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6979.295534] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6982.671222] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 03:17:39 (1784099859) [ 6983.271385] Lustre: 252528:0:(genops.c:1795:obd_export_evict_by_uuid()) lustre-MDT0000: evicting fc790c88-3fb0-4a73-ad90-cb2e5c02ba96 at adminstrative request [ 6988.002290] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 03:17:45 (1784099865) [ 6989.761727] Lustre: Failing over lustre-MDT0000 [ 6989.958990] Lustre: server umount lustre-MDT0000 complete [ 7002.797497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.193154] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7008.274894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14517 to 0x2c0000401:14561) [ 7008.274917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14548 to 0x280000401:14593) [ 7010.016672] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7010.507739] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7013.804661] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 03:18:10 (1784099890) [ 7017.472701] Lustre: Failing over lustre-OST0000 [ 7019.537387] Lustre: server umount lustre-OST0000 complete [ 7032.115081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7034.014374] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7037.614732] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7038.157582] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7042.022209] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 03:18:39 (1784099919) [ 7043.421411] Lustre: Failing over lustre-MDT0000 [ 7043.686468] Lustre: server umount lustre-MDT0000 complete [ 7046.522282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7048.092169] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7048.884575] Lustre: lustre-MDT0000: Denying connection for new client 1f94c39e-b022-4515-991b-7a8cd355a332 (at 192.168.201.12@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 7048.889577] Lustre: Skipped 2 previous similar messages [ 7051.765423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14548 to 0x280000401:14625) [ 7051.765428] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14517 to 0x2c0000401:14593) [ 7056.176758] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 03:18:53 (1784099933) [ 7058.966788] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7059.836220] Lustre: Failing over lustre-OST0000 [ 7059.870043] Lustre: server umount lustre-OST0000 complete [ 7074.777834] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7074.779425] LDISKFS-fs (dm-2): recovery complete [ 7074.783733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7077.376024] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7081.167253] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7081.706167] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7086.086253] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 03:19:23 (1784099963) [ 7089.535865] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7091.471277] Lustre: Failing over lustre-OST0000 [ 7091.508251] Lustre: server umount lustre-OST0000 complete [ 7105.721604] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7105.722992] LDISKFS-fs (dm-2): recovery complete [ 7105.726249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7107.646810] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7107.700558] LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.201.12@tcp inode [0x2000301a1:0x5:0x0] object 0x280000401:14627 extent [0-4194303]: client csum 927b3c05, server csum e54c0576 [ 7111.501790] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7112.053925] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7115.520030] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 03:19:52 (1784099992) [ 7118.049161] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 7120.542182] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 7122.833132] Lustre: Failing over lustre-MDT0000 [ 7123.046052] Lustre: server umount lustre-MDT0000 complete [ 7124.376596] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784100001 with bad export cookie 9498434355229668160 [ 7124.378760] Lustre: Failing over lustre-OST0000 [ 7124.380902] LustreError: 6484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7124.428953] Lustre: server umount lustre-OST0000 complete [ 7140.196592] LDISKFS-fs (dm-0): 2 truncates cleaned up [ 7140.198348] LDISKFS-fs (dm-0): recovery complete [ 7140.201100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7149.025348] LustreError: 3627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9b3ad12d6d80 x1870757088912768/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7150.870834] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7154.263097] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14517 to 0x2c0000401:14625) [ 7166.240091] LDISKFS-fs (dm-2): 3 truncates cleaned up [ 7166.241659] LDISKFS-fs (dm-2): recovery complete [ 7166.246156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7168.294721] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7176.178001] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 03:20:53 (1784100053) [ 7181.056040] Lustre: Failing over lustre-OST0000 [ 7181.110610] Lustre: server umount lustre-OST0000 complete [ 7182.435116] Lustre: Failing over lustre-MDT0000 [ 7182.745812] Lustre: server umount lustre-MDT0000 complete [ 7196.310408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7197.897855] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7201.804519] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14517 to 0x2c0000401:14657) [ 7206.232374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7208.518383] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7209.292716] Lustre: lustre-OST0000: Denying connection for new client 0e45c0cc-716b-4cd9-8479-6ee28831e737 (at 192.168.201.12@tcp), waiting for 3 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:09 [ 7214.349544] Lustre: lustre-OST0000: Denying connection for new client 0e45c0cc-716b-4cd9-8479-6ee28831e737 (at 192.168.201.12@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 7224.589409] Lustre: lustre-OST0000: Denying connection for new client 0e45c0cc-716b-4cd9-8479-6ee28831e737 (at 192.168.201.12@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 7224.598129] Lustre: Skipped 1 previous similar message [ 7245.069180] Lustre: lustre-OST0000: Denying connection for new client 0e45c0cc-716b-4cd9-8479-6ee28831e737 (at 192.168.201.12@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:33 [ 7245.078746] Lustre: Skipped 3 previous similar messages [ 7278.500123] Lustre: lustre-OST0000: recovery is timed out, evict stale exports [ 7278.502148] Lustre: 266054:0:(genops.c:1622:class_disconnect_stale_exports()) lustre-OST0000: disconnect stale client 1f94c39e-b022-4515-991b-7a8cd355a332@ [ 7278.505339] Lustre: 266054:0:(genops.c:1622:class_disconnect_stale_exports()) Skipped 1 previous similar message [ 7278.507591] Lustre: lustre-OST0000: disconnecting 1 stale clients [ 7278.515766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14676 to 0x280000401:14697) [ 7282.092885] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 69 sec [ 7290.286862] Lustre: DEBUG MARKER: free_before: 7646268 free_after: 7646268 [ 7292.723013] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 03:22:49 (1784100169) [ 7294.480229] Lustre: Failing over lustre-OST0000 [ 7294.516765] Lustre: server umount lustre-OST0000 complete [ 7308.648982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7311.358571] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7316.627942] Lustre: DEBUG MARKER: == replay-single test 93a: replay + reconnect ============ 03:23:13 (1784100193) [ 7318.357047] Lustre: Failing over lustre-OST0000 [ 7318.413206] Lustre: server umount lustre-OST0000 complete [ 7331.630383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.094892] LustreError: 269247:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 40000ms [ 7333.099943] LustreError: 269247:0:(ldlm_lib.c:2904:target_recovery_thread()) Skipped 79 previous similar messages [ 7334.313200] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7339.488198] Lustre: *** cfs_fail_loc=715, val=40*** [ 7348.492603] Lustre: lustre-OST0000: Client 0e45c0cc-716b-4cd9-8479-6ee28831e737 (at 192.168.201.12@tcp) reconnected, waiting for 3 clients in recovery for 1:25 [ 7349.217084] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:24 [ 7349.221910] Lustre: Skipped 1 previous similar message [ 7354.848181] Lustre: *** cfs_fail_loc=715, val=40*** [ 7354.851093] Lustre: Skipped 2 previous similar messages [ 7364.577598] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnected, waiting for 3 clients in recovery for 1:08 [ 7364.581682] Lustre: Skipped 1 previous similar message [ 7370.720183] Lustre: *** cfs_fail_loc=715, val=40*** [ 7370.722853] Lustre: Skipped 2 previous similar messages [ 7373.144118] LustreError: 269247:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7373.147529] LustreError: 269247:0:(ldlm_lib.c:2904:target_recovery_thread()) Skipped 78 previous similar messages [ 7375.232478] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7375.827879] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7379.939867] Lustre: DEBUG MARKER: == replay-single test 93b: replay + reconnect on mds ===== 03:24:16 (1784100256) [ 7381.554160] Lustre: Failing over lustre-MDT0000 [ 7381.795650] Lustre: server umount lustre-MDT0000 complete [ 7385.057201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.058437] LustreError: 266508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7385.061875] Lustre: Skipped 52 previous similar messages [ 7385.068043] LustreError: 266508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 229 previous similar messages [ 7394.897696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7395.077060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7395.079867] Lustre: Skipped 19 previous similar messages [ 7395.095277] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7395.101060] Lustre: Skipped 19 previous similar messages [ 7395.597362] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7395.603237] Lustre: Skipped 19 previous similar messages [ 7396.724947] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 7400.426584] LustreError: 270801:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 sleeping for 80000ms [ 7401.952185] Lustre: *** cfs_fail_loc=715, val=80*** [ 7401.954949] Lustre: Skipped 2 previous similar messages [ 7406.560258] Lustre: *** cfs_fail_loc=715, val=80*** [ 7411.993411] Lustre: lustre-MDT0000: Client 0e45c0cc-716b-4cd9-8479-6ee28831e737 (at 192.168.201.12@tcp) reconnected, waiting for 2 clients in recovery for 1:28 [ 7411.999157] Lustre: Skipped 1 previous similar message [ 7415.777663] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7418.336173] Lustre: *** cfs_fail_loc=715, val=80*** [ 7428.364644] Lustre: lustre-MDT0000: Client 0e45c0cc-716b-4cd9-8479-6ee28831e737 (at 192.168.201.12@tcp) reconnected, waiting for 2 clients in recovery for 1:12 [ 7432.160728] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7434.720200] Lustre: *** cfs_fail_loc=715, val=80*** [ 7434.721576] Lustre: Skipped 1 previous similar message [ 7443.724972] Lustre: lustre-MDT0000: Client 0e45c0cc-716b-4cd9-8479-6ee28831e737 (at 192.168.201.12@tcp) reconnected, waiting for 2 clients in recovery for 0:56 [ 7448.544762] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7460.109091] Lustre: lustre-MDT0000: Client 0e45c0cc-716b-4cd9-8479-6ee28831e737 (at 192.168.201.12@tcp) reconnected, waiting for 2 clients in recovery for 0:40 [ 7464.928687] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID [ 7471.072197] Lustre: *** cfs_fail_loc=715, val=80*** [ 7471.074145] Lustre: Skipped 4 previous similar messages [ 7480.512127] LustreError: 270801:0:(ldlm_lib.c:2904:target_recovery_thread()) cfs_fail_timeout id 715 awake [ 7480.520218] Lustre: 270801:0:(ldlm_lib.c:2950:target_recovery_thread()) too long recovery - read logs [ 7480.520258] Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7480.523043] LustreError: dumping log to /tmp/lustre-log.1784100357.270801 [ 7480.524880] Lustre: Skipped 56 previous similar messages [ 7480.605808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:14711 to 0x280000401:14729) [ 7480.605813] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:14670 to 0x2c0000401:14689) [ 7482.611376] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7483.098812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7487.180352] Lustre: DEBUG MARKER: == replay-single test complete, duration 7333 sec ======== 03:26:04 (1784100364) [ 7487.980357] Lustre: DEBUG MARKER: === replay-single: start cleanup 03:26:04 (1784100364) === [ 7491.763913] Lustre: DEBUG MARKER: === replay-single: finish cleanup 03:26:08 (1784100368) === [ 7525.345956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7525.347709] Lustre: Skipped 20 previous similar messages [ 7528.171175] Lustre: server umount lustre-MDT0000 complete [ 7531.537618] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784100409 with bad export cookie 9498434355229680921 [ 7531.542826] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7531.549329] LustreError: 6482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7531.557074] LustreError: Skipped 7 previous similar messages [ 7531.689560] Lustre: server umount lustre-MDT0001 complete [ 7544.844362] Lustre: server umount lustre-OST0000 complete [ 7557.673850] Lustre: server umount lustre-OST0001 complete [ 7564.148431] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 7565.532172] Key type lgssc unregistered [ 7565.698564] LNet: 274092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7565.703403] LNetError: 274092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7565.712518] LNet: Removed LNI 192.168.201.112@tcp [ 7566.161206] Key type .llcrypt unregistered [ 7566.163193] Key type ._llcrypt unregistered